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ABSTRACT 


The Department of Defense has recently declared that irregular warfare is as 
strategically important as traditional warfare. Unfortunately, there is a dearth of mature 
training and analysis tools that can be used to support contemporary military operations. 
One popular wargaming simulation is the campaign-level Peace Support Operations 
Model (PSOM). This thesis provides a quantitative analysis of PSOM. The results are 
based on over 75,000 simulated runs of an Operation Iraqi Freedom scenario. The study 
concludes with the identification of the critical factors within PSOM, recommended 
potential uses for the model, an accuracy assessment, and an assessment of the risks 
assumed by using the model. Results indicate that the critical factors within the model 
are indicative of contemporary operations. _PSOM should be used for its original 
purpose, as a wargame to further study the societal implications of modern military 
operations. As a wargame, PSOM has strong potential as a high-level staff and leader 
training tool and as a planning aid for course of action development. Within the confines 
of this study, the model proved limited in its ability to model changes in force 
capabilities. Due to its limited ability to model uncertainties in irregular warfare without 
the human-in-the-loop, or give multiple potential outcomes, further development and 


analysis is required before the model is used for large scale analysis. 
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EXECUTIVE SUMMARY 


This research addresses the Department of Defense’s (DoD’s) need to accurately 
model strategic level contemporary military operations. In the simplest form, the purpose 
is to gain a basic understanding of the Peace Support Operations Model (PSOM). In 
doing so we provide a methodology which should be incorporated into the VV&A 
(verification, validation, and accreditation) process for complex combat models that 
incorporate the Political, Military, Economic, Social, Infrastructure, and Information 
(PMESIJ) aspects of irregular warfare. This summary explains the need for such a model 
and hence the methodology, the steps taken during the analysis of PSOM, and the 
conclusions and recommendations toward the future use of PSOM and the study of 
PMESII models. The following document provides the DoD insights into the potential 


use of PSOM as a wargame and as an analytic tool. 


As the United States Military’s focus shifts from conventional warfare toward 
irregular warfare, interest has progressively grown in the development of models that can 
simulate social behavior as it pertains to military operations. Populations, whether 
broken into smaller social groups, granulated into individuals, or studied as an aggregate 
of social groups, are often the determinate of success in modern combat. According to 
the most recent U.S. Army doctrine: 

The integration of civilian and military efforts is crucial to successful 

COIN [Counter Insurgency] operations. All efforts focus on supporting 

the local populace and the H[ost] N[ation] government. Political, social 

and economic programs are usually more valuable than conventional 


military operations in addressing the root causes of conflict and 
undermining an insurgency. FM 3—24 Counterinsurgency 


The military uses models for course of action analysis, training and rehearsal, and 
evaluation for acquisition. If these models are not indicative of contemporary operations, 
they are not only lacking in utility, they are potentially harmful. Therefore, the military’s 
interest in modeling social cognition has grown out of necessity. To date there has not 
been a validated model designed for irregular warfare that covers the instruments of 


national power: Diplomatic, Information, Military, Economic (DIME) or the Political, 
XIX 


Military, Economic, Social, Infrastructure, and Information (PMESII) indicators on 
which progress in irregular warfare is based. Figure 1 illustrates the complex causal 
relationship between DIME and PMESII factors. As one can imagine, the development 
of such a model is not trivial. According to the Defense Modeling and Simulation 
Analysis Committee, the data to instantiate such a model is either nonexistent or woefully 


inaccurate, and the validation process of such a model would have to be completely 
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Figure 1. The complexity of the PMESII Environment [From (Allen 2004)] 


One new model that addresses the operational focus on the population is the 
Peace Support Operations Model (PSOM). PSOM is a campaign level, simulation based, 
human-in-the-loop wargame which portrays the civilian populace as agents within the 
model. The model’s metrics are both quantitative and qualitative in nature, providing 
results which are analogous with irregular warfare. Many of the Measures of 
Effectiveness offered by PSOM, such as attrition, are easily understood. However, much 
of the output is a cultural representation of the uncertain effects military and political 
actions have on the population. 
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This analysis defines a logical methodology to assess PSOM as a potential tool to: 


e Quantitatively measure the limitations and constraints of PSOM 
and, more importantly, identify the appropriate context for 
interpreting PSOM results. 


e Assess the accuracy of PSOM in regards to current doctrine. 
e Make recommendations toward the potential use of PSOM. 


e Define a methodology for the much needed VV&A process of PMESII 
models. 


In order to address the fore mentioned issues, this study follows the data farming 
process (Horne 2004). Our implemented data farming process entails a four step 
procedure: Define factors of interest, create a design of experiment (DOE), run the 
simulation experiments in parallel on a computing cluster, and conduct data mining. By 
leveraging thoughtful experimental design with powerful computation capabilities we are 
able to change over 100 parameters within PSOM while conducting over 50,000 
simulations of operations in the current Multi-National Forces Iraqi Area of 
Responsibility. This immense number of simulation runs allows us to quantifiably 
analyze over 5,000 design points consisting of well over 500,000 data files covering a 
large portion of the model’s response space. By covering such a vast space of possible 


outcomes, we are able to use a variety of methods to investigate the model’s behavior. 


The analysis of this data set focused on three doctrinally essential measures of 
effectiveness (MOEs) provided by PSOM in a current Iraq based scenario. These 
measures are the Sunni Population’s consent toward the Coalition forces, the Sunni 
Population’s consent toward the Iraqi Government, and the security level of the nation. 


The results of our research are telling. 


PSOM provides players a tremendous amount of flexibility in choosing their 
operational tasks (stances). Our analysis shows that the stances players (factions) choose 
have a significant effect on both the consent and security of the population, which is what 
one would expect based on current doctrine. There are also intuitive interactions between 
the stances of multiple factions. The rules of engagement status and risk level of a unit in 


conjunction with the unit’s stance also prove important choices throughout the game. 
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For example, a violent ROE in conjunction with an aggressive stance will result in a loss 
of consent, especially if another faction proves more cautious and focused on the 


population. 


We found the consent metric to be more intricate than the security metric. It 
seems consent is difficult, but not impossible, to increase toward both the coalition and 
the Iraqi government. It is arguable that this difficulty is actually a strength of the game. 
Changing the opinion of a society is a particularly difficult task; just as in PSOM, it can 
be done, but not without deliberate effort. 


The initial conditions and assumptions made in scenario design are of particular 
importance. The simulation’s results can be dominated by some of these assumptions. 
For example, the underlying assumptions that are used to create the population are 
absolutely critical in the determination of consent in the game. In accordance with 
doctrine and the PSOM developers, the outcomes of PSOM are especially population 
centric. Because of this, if underlying assumptions about the population are not strongly 
supported, the game should be played multiple times, changing population assumptions 


to ensure a considerable range of possible outcomes is covered. 


The game is not very sensitive to many of the non-scenario specific parameter 
settings. Of particular focus within this study are those of unit capabilities. Therefore, 
we recommend PSOM should not be used as a decision tool for equipment manning or 
force manning until this aspect of the game is studied further. Those settings to which we 


found the game to be sensitive are mentioned within the thesis. 


The model should be used for its original purpose, as a wargame to further study 
the societal implications of modern military operations. As a wargame, PSOM has strong 
potential as a high-level staff and leader training tool and as a planning aid for course of 
action development for stability operations. Within the confines of this study, the model 
proved limited in its ability to model changes in force capabilities. Also, due to its 
inability to model uncertainties in irregular warfare, or give multiple potential outcomes 
further development and analysis is required before the model is used for large scale 


analysis. 
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I. INTRODUCTION 


At an earlier time, a commander could be certain that a future war would 
resemble past and present ones. This enabled him to analyze appropriate 
tactics from past and present. The troop commander of today no longer 
has this possibility. He knows only that whoever fails to adapt the 
experiences of the last war will surely lose the next one. 


German Gen. Franz Uhle-Wettler (1985) 


A. OVERVIEW 


As the focus of the United States military shifts from conventional warfare toward 
irregular warfare, interest has progressively grown in the development of models that can 
simulate social behavior as it pertains to military operations. The contemporary 
operating environment, as reflected in Iraq and Afghanistan, shows the critical role the 
population plays in modern combat. Populations, whether broken into smaller social 
groups, granulated into individuals, or studied as an aggregate of social groups, are often 
the determinate of success in modern combat. Therefore, the military’s interest in 
modeling social cognition has grown out of necessity. The military uses models for 
course of action analysis, training and rehearsal, and evaluation for acquisition. If these 
models are not indicative of contemporary operations, they are not only lacking in utility, 


but are potentially harmful. 


One new model that combines conventional warfare with the modern focus on the 
population is the Peace Support Operations Model (PSOM). PSOM is a simulation-based 
wargame, that portrays the populace and displays the effects military and political actions 
have on the population and its emersion of social organizations. During preliminary use 
of PSOM at the Joint Staff Warfighting Analysis Division and the Office of the Secretary 
of Defense, PSOM has shown potential as an analytic and training tool; however, to date 
the model has not been taken through any sort of verification, validation and accreditation 
(VV&A) process. VV&A is an important prerequisite, ensuring models with flaws or 
biases to not become mainstream tools. The lack of appropriate vetting can prove 


detrimental due to the tremendous risk inherited from using a model which may provide 
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inconsistent or inaccurate results. This study quantifiably analyzes PSOM using design 
of experiments and data farming to quantitatively measure the limitations and constraints 


of the model. 
B. BACKGROUND 


Warfare changes with society. Over the past fifty years we have seen a shift from 
maneuver warfare where nonlinear tactics could determine victory (German blitzkrieg), 
to irregular warfare where the objective is the support of a population (Mao Tse-tung and 
the Protracted People's War). Irregular warfare is a violent struggle among state and non- 
state actors for legitimacy and influence over the relevant population (Department of 
Defense [DoD] Directive 3000.07, 2007). The Department of Defense’s policy is to 
“Recognize that IW [irregular warfare] is as strategically important as traditional 
warfare” (DoD, 2007). Irregular warfare contains many subsets, including 
unconventional warfare, foreign internal defense, counter insurgency (COIN), 


counterterrorism, and stability operations. 


The United States military continues to transform to meet the current and 
upcoming challenges of national security. Although the United State military must 
remain able to defend the country and defeat a robust enemy in a conventional war, the 
military now must conduct the complete taxonomy of irregular warfare to ensure 
America’s security. In doing so, the military focus is split between the destruction of 
enemy armies and the development of “indigenous capacity for securing essential 
services, a viable market economy, rule of law, democratic institutions, and robust civil 
society” (DoD Directive 3000.05, 2005); the latter obviously being the far more difficult 
to plan. As C.E. Callwell states in Small Wars (1906): 

But when there is no king to conquer, no capital to seize, no organized 

army to overthrow, and when there are no celebrated strongholds to 


capture, and no great centers of population to occupy, the objective is not 
so easy to select. (p. 40) 


Colonel Callwell continues by explaining how an attacking force must destroy 
and deprive the defending population using techniques that are inhumane and not 
accepted under today’s laws of war. Contemporary doctrine enforces the mitigation of 


civilian casualties and collateral damage. 


The role of the military has become extremely complex, interwoven with other 
governmental and non-governmental agencies. Current war spans political, economic, 
social and military aspects of humanity (Hammes, 2008). The military has published 
new doctrine and leaders have been quick to change their mindset. As young men and 
women gain experience in this emerging warfare, it has become the norm rather than the 
exception for warfighters to leverage the political, economic, social, and military aspects 
of war. The result is a military composed of people ready to meet the irregular warfare 
challenges and accomplish the mission in stride. This trend will continue as junior 
leaders become senior leaders with a wealth of modern-day experience and knowledge 


which will in turn become wisdom. 


As the paradigm of the military’s role changes, the plans, training, and force 
structure must be dynamic as well. Unfortunately, a key part of the planning process, 
training structure, and military decision making process has yet to be transformed. This 
key ingredient is the modeling and simulation of war. Whether manifested in a wargame, 
simulation, or experiment, simulation modeling has become a critical part of the decision 
making process. An example is the critical role that modeling takes in DoD’s 


development of the Analytic Agenda (Stevens, 2003). 


According to the Committee on Modeling and Simulation for Defense 
Transformation, “DoD needs MS&A appropriate to complex dynamic, adaptive systems 
because such systems pervade military combat, other aspects of military operations, and 
other political, military, economic, social, infrastructure, and information phenomena of 
interest” (Committee on Modeling and Simulation for Defense Transformation, 2006). 
DoD’s Transformation Planning Guidance (2003) states “a new generation of M&S is 
needed to support concept development linking together many types of simulations from 
aggregate and detailed computer models to simulators and man in the loop hardware 


components.” 


The Department of Defense defines validation as the process of determining the 
degree to which a model, simulation, or federation of models and simulations, and their 
associated data are accurate representations of the real world from the perspective of the 
intended use(s) (DoD, 2008). To date there has not been a validated model designed for 
irregular warfare that covers the instruments of national power, Diplomatic, Information, 
Military, Economic (DIME) or the Political, Military, Economic, Social, Infrastructure, 
and Information (PMESIJ) indicators that progress in irregular warfare is based. Figure 2 
illustrates the complex causal relationship between DIME and PMESII factors. As one 
can imagine, the development of such a model is not trivial. According to the Defense 
Modeling and Simulation Analysis Committee, the data to instantiate such a model is 
either nonexistent or woefully inaccurate, and the validation process of such a model 
would have to be completely rethought (Committee on Modeling and Simulation for 


Defense Transformation, 2006). 
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Figure 2. The Complexity of the PMESII Environment [From (Allen, 2004)] [Best 
viewed in color] 


Gropman (1986) states a popular position in the development of mathematical 
models pertaining to human social interaction as: 

Psychological dimensions: their uncertainty, their variety, their 

inconsistency and their lack of utility in modeling future conflict should 


make one reluctant to expend resources pursuing them, leaving the 
community the time and money to quantify better the quantifiable. (p. 16) 


When modeling warfare, especially on a macro level, much of the information 
about human behavior is either assumed or discarded, often resulting in a gross 
simplification of the situation we wish to model (Perla, 1990, p. 276). This problem is 
amplified when attempting to model irregular warfare. In FM 3-07, the newest Army 
Manual on Stability Operations, LTG Caldwell states: 

The lines separating war and peace, enemy and friend have blurred and no 

longer conform to the clear delineations we once knew. At the same time, 

emerging drivers of conflict and instability are combining with rapid 


cultural, social, and technological change to further complicate our 
understanding of the global security environment. (Army, 2008) 


This statement alone depicts the modeling of irregular warfare as a great problem to even 


comprehend let alone attack. 


Our military has become dependent on models. Traditionally the military has 
used computational models for analysis and forecasting for planning, simulation for 
training rehearsal, and design and evaluation for acquisition (Committee on 
Organizational Modeling, 2008, p. 23). However, we are now fighting a war that many 
deem extremely difficult to model, much less model in a way that would fit traditional 
validation techniques. Due to the dependence on models and simulation, the DoD has 
attacked this problem with vigor, and the modeling community has answered the call to 


develop what are now being called PMESII models. 


There are many models being developed to meet the need to understand the 
contemporary battlefield. Currently these models fall into multiple categories such as 
agent based, system dynamic, or analytic. These models differ in their underlying 


designs used to generate outputs. However, they are similar in that they are a step in the 


direction of modeling the asymmetric battlefield where “civilians are the targets, 
objectives to be won, as much as an opposing force” (Smith, 2007). One such model is 


the Peace Support Operation Model v2 (PSOM 2). 


PSOM was developed by the United Kingdom (UK) to first study and then 
understand stabilization operations. Model development began in 2004 at the Defense 
Science and Technology Laboratory, United Kingdom. The model is built by an allied 
institution which has already encouraged inter-agency use of the simulation. The model 
is a time stepped human-in-the-loop semi-automated campaign level wargame. PSOM 
uses over eighty algorithms, which are primarily deterministic with a number of 
stochastic elements, to take into account the DIME interventions and PMESII indicators 
in determining outcomes of friendly unit actions (Body, 2008). These outcomes are 
representative of required results from irregular warfare where seized terrain and 


causality counts cannot determine a victor or even positive results. 


A critical underlying assumption of PSOM is that current UK and U.S. doctrine 
represent the best strategy in Peace Support Operations. This assumption is difficult to 
validate due to the changing nature of irregular warfare, and doctrine is being developed 
and improved on a regular basis. However, the developers of the model support this 
assumption stating that the contemporary environment is being debated and that PSOM is 
“an endorsed understanding of the Contemporary Operating Environment” (Body, 2008). 
This statement is not meant to be a catch-all for every assumption in which the model is 
based, but rather a candid explanation of those areas of “deep uncertainty,” which can 


derive from social modeling. 


The PSOM 2 model allows for the interaction of multiple factions amongst 
themselves and the population. This is key because the model assumes that the 
Contemporary Operating Environment (COE) is one in which the population is the point 
of emphasis. The modelers understand that irregular warfare uses a range of approaches 
including political, military, economic, and social to persuade the enemy leadership that 
their strategic goals are either unachievable or not cost effective. Therefore, a 
tremendous emphasis in PSOM is its ability to represent both the political will of the 


population and the changes both friendly and enemy forces have on this will. In order to 
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model these interactions, PSOM allows each player or faction to choose a stance for each 
time step. The stances are similar to current doctrinal operations and specific tactical 
tasks and are directed toward other factions or the population. For example, a maneuver 


battalion in Iraq could have the stance of secure being directed at the Sunni Population. 


PSOM gives multiple measures of effectiveness (MOEs) and metrics that can be 
used to determine success or lack thereof. The model is analogous with U.S. doctrine in 
that political legitimacy is a key outcome of stability operations. Legitimacy is assessed 
by the indicators of security, consent, rule of law, and provision of essential services. The 
primary MOEs presented by PSOM are security, consent, stability and fear. These MOEs 
coincide with the strategic framework described in FM 3-07, which states the end state 
conditions for a stability operation are a safe and secure environment, established rule of 
law, social well being, stable governance and a sustainable economy. Although economy 
is not a stated MOE in PSOM, economic conditions play a role in all the aforementioned 


MOEs and multiple economic metrics are available. 


For all of its strengths, to date this model has not been through any particular 
VV&A process. The J—8 Warfighting Analysis Division along with the Office of 
Security of Defense Simulation Analysis Center (OSD-SAC) have used PSOM in an 
exploratory manner with debatable results. Currently, the staff at DSTL are conducting a 
large-scale multiple course of action wargame to test PSOM. However, PSOM has not 


been put through the rigors of the DoD’s Modeling and Simulation accreditation process. 
C. RESEARCH QUESTIONS 


The intent of this research is to conduct a quantitative analysis of the Peace 
Support Operations Model. This analysis is not meant to be a complete VV&A process. 
However, the following issues are addressed: 

e Identify the factors which most dramatically change PSOMs output. 

Identifying the critical factors quantitatively measures the limitations and 


constraints of PSOM, and, more importantly, identify the appropriate 
context for interpreting PSOM results. 


e Attempt to assess the accuracy of the Peace Support Operation Model. 


e Make recommendations toward the potential use of the Peace Support 
Operation Model. 


e Conduct a risk analysis of the Peace Support Operation Model. 


D. BENEFITS OF STUDY 


This study provides the Department of Defense a thorough analysis of PSOM. In 
doing so it aids decision makers in selecting potential uses of PSOM as a wargame and 
analytic tool. Concurrently, we have developed an analytic procedure for evaluating 
complex PMESII models. This procedure provides analysts a reference in the validation 
and verification of models that take into account the importance of societal implications. 
This study demonstrates that the use of large scale data farming, combined with 
thoughtful design of experiments, can provide useful insight into the complexities of 


social modeling. 
E. METHODOLOGY 


This thesis uses quantitative analysis to explore the capabilities of PSOM. 
Because the parameter space in a campaign level model such as PSOM is quite large, the 
study is based on the leveraging of high performance computing and efficient design of 
experiments to run the model many times. This process allows for the exploration of a 
very large parameter space in a limited amount of time (Barry and Koelher, 2004). 
Efficient design of experiments and statistical analysis permits us to determine which 
parameters and interactions are significant in PSOM and what the corresponding 
responses are to a particular set of parameters. Once the significant factors are identified, 
further exploration into particular sets of parameters permits analysis of the response 


accuracy in accordance with current doctrine. 


The scenario used to test the model is the ongoing war in Iraq as of 2004. This 
model was developed by DSTL in 2008 and has been vetted through multiple U.S. Army 
officers who served in the Iraqi theater during this period of time. The design of 
experiments focus on the underlying assumptions about the Iraqi population, the 
capabilities and attributes of coalition and insurgent forces, the operational courses of 


action taken by coalition forces, and the systematic settings of PSOM. The responses 
8 


analyzed are primarily the changes in security in the nation and the population’s consent 
towards its own government and coalition forces (when needed, other outputs are taken 
into account). The resulting statistical analysis of the simulated data is then used to gain 


insight into the vast space of possible PSOM inputs and their corresponding outputs. 


THIS PAGE INTENTIONALLY LEFT BLANK 
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HW. PSOM BACKGROUND 


The modeling of cognition and action by individuals and groups is quite 
possibly the most difficult task humans have yet undertaken. 


(Committee on Organizational Modeling, 2008, p. 20) 


A. BACKGROUND 


This next chapter is a combination of first person user experience with the model, 
the 2008 PSOM 2 Functional Specifications, and the 2008 PSOM Philosophy. The 
purpose of this chapter is to provide the reader a basic understanding of the PSOM model 
and software. This is not meant to be a standalone user’s manual, but rather an 
introduction to the framework of the model. By understanding many of the 
specifications, the data required, the setup of the model, and the execution of the 
simulation the reader can see not only what the model is intended to do, but how it 


accomplishes it. 


The Peace Support Operations Model is a human-in-the-loop, time stepped semi- 
automated wargame (Parkman, 2008). The definition of a wargame is a model or 
simulation not involving actual military forces, in which the flow of events is affected by, 
and in turn affects, decisions made during the course of those events by players 
representing opposing sides (Perla, 1990, p. 274). PSOM is a campaign level model that 
represents irregular warfare at the Policy, Strategic and Operational Levels (Body, 2008). 
It was designed by and is still in active development at the Defense Science and 
Technology Lab of the UK Ministry of Defense in order to test policy guidance and 
provide campaign context for lower level modeling. PSOM 2 models irregular warfare as 
defined by the Department of Defense. Because the results of an irregular warfare 
campaign rest on the will of the population, PSOM shows the causal effects of the 
players’ actions primarily through their effect on the population. The model is analytical 
and based on the assumption that current U.S. and UK irregular warfare doctrine 


represent the solutions to irregular warfare (Body, 2008). The designers acknowledge the 
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significance of this assumption, but also accept that there is no universally accepted 
validated answer to irregular warfare. So, until combat models can be tied into an agreed 


upon anthropological model, DSTL has moved forward with PSOM. 


PSOM is written in Visual Basic and is non-proprietary. For this study and future 
studies, analysts are not just allowed to “look under the hood” but encouraged to do so. 
PSOM is built from a series of algorithms and sub-models which are interwoven to 
represent the DIMEFIL and PMESII aspects of irregular warfare. In this aspect it is 
possible to classify PSOM as a system dynamics model. In addition, the emersion of 
social groups within the civilian population is portrayed by independent agents whose 
actions and attitudes are determined by simple decision rules, thus giving PSOM a 
pseudo agent based characterization. One key aspect is PSOM supports the interaction 
of multiple organizations that are representative of IW. PSOM uses over eighty 
algorithms to drive the unit interaction and state functionality systems. These 
interactions and functionality systems allow for 3™ and 4" level actors, such as non- 


governmental organizations, to have effects on outcomes. 
B. SPECIFICATIONS AND CAPABILITIES OF PSOM 


The PSOM wargame is designed with a two-level hierarchy: the high level game 
(HLG) and the operational game. The high level game is designed to simulate the 
political and strategic levels of conflict. This is where relationships between factions are 
represented and international cooperation is accounted for. The high level game is very 


much the “grey beard” game. 


The operational level game is just that, operational. It is representative of the 
campaigns and actions required to support the strategic objectives. This resolution is the 
level at which Brigade Combat Teams, terrorist organizations, and local government 
organizations are played. By creating this ontology the developers have attempted to 
represent the importance of the political and strategic influences with regards to the 


operational levels of irregular warfare. 


The PSOM 2 wargame model can be split into four categories: scenario design, 


underlying data and settings, game play, and results. 
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C; SCENARIO DESIGN 


In the development of social models the represented environment is dynamic and 
the model’s structure is often derived from logical, rather than only mathematical, 
specifications (Gilbert, 2005, p. 15). PSOM’s scenario design accounts for this dynamic 
behavior. The scenario design is the point at which the modeler develops the specific 
initial conflict setting. In a basic wargaming analogy, this is where we pick the layout of 
the game board, the attributes of the pieces, and the mindset of the population. Many of 
the basic assumptions about the social model are inputted into the scenario design. An 


abridged version of the basic inputs required for the scenario follows. 
1. Factions 


A faction is defined as any political entity that has an effect on the scenario. 
These are the key players in the game. Faction examples are military organizations, the 
host nation government, NGO’s, and terrorist organizations. A significant characteristic 
in the makeup of a faction is its ideology. Factions have ideologies represented by the 
Nolan Chart, which uses scores for economic and personal freedom to determine the 
faction’s political and social inclination. Figure 3 shows the Nolan Chart and gives the 
explanation from the PSOM Manual. Other inputs for each faction are type of unit or 
organization (there are many pre-set unit types), size in number of people, expectations, 


wealth level, etc. 


The Nolan chart is used to determine a faction’s political views. A faction is 
assigned a number based on the importance of political freedom versus economic 
freedom. Where prior tools determined if an individual’s views were to the right or left 
on a line, the Nolan chart is a plane. The Nolan chart was developed by the libertarian 
party and its role in PSOM has drawn criticism due to its bias and lack of scientific 


support (Turnley, 2008). 
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Figure 3. An Example of a Nolan Chart Taken from the PSOM User Manual 
(Draft) 


2. Ethnic Groups 


This category represents the attributes of groups of similar people within the 
population. For the model, ethnic groups can be used for more than just ethnicities, but 
also social views (Parkman, 2008). Ethnic groups also use the Nolan Chart to determine 
political ideology. However, ethnic groups are assigned marginal gains coefficients. 
Marginal gains coefficients represent the importance of a particular good or service (to 
include security) to the corresponding ethnicity. These marginal gain coefficients are 
used to determine consent in the operational game. Other inputs into the ethnic group 
category are starting values of population size, age groups, etc. As the aggregate of 
ethnic groups create a population agent, the assumptions that create an ethnic group 


should prove to be of the utmost importance. 
nF Nationalities 


Every faction has a nationality. This attribute allows the game to look at the 


homeland consent based on activities particular to the individual nations. 
4. Map 


The map plays an important role in the PSOM. The user downloads the map of 


the area of concern. Then the map is divided by grid squares representing a user 
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determined geographic size. The user then gives each square particular attributes based 


on physical terrain data, population level, infrastructure level, and human capital. 
5. Population Agents 


By using the input of map data and ethnic group data, population agents are 
formed. These are “Semi Autonomous groups of ‘similar’ people of a defined size based 
on proximity. An agent may make a number of decisions based on the conditions and 
environment surrounding it” (Parkman, 2008, p. 41). The population agents gain the 


attributes of their respective ethnic group with a predetermined user defined variance. 
6. Relationships 


PSOM allows the user to define overarching relationships between factions. 
These relationships can be updated throughout the game, but are initialized in the 


scenario design. The relationships are depicted below in Table 1. 

















Combative Relationship Intelligence Relationship 
Shooting Sharing 

Shooting Not Sharing 

Not Shooting Sharing 

Not Shooting Not Sharing 














Table 1. Explanation of Relationships in PSOM 


7. Units 


These are the conventional military forces, nonconventional military forces, 
insurgent or terrorist forces, and governmental and non-governmental agencies. The user 
can give units particular leadership values, experience values, stances, etc. This is where 
you build the hierarchy of units and commanders and assign the units to their initial 


location. This also allows for the building of battle groups. 
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D. DATA AND SETTINGS 


The underlying data and settings for the model establish the basic characteristics 
and assumptions that can be generalized over multiple scenarios. These characteristics 
are predominately independent of the scenario, but can be updated to reflect scenario 


specific requirements. 
1. Unit Abilities 


Under data and settings, the user is able to prescribe unit abilities. This category 
allows the user to assign both descriptive and subjective values to multiple attributes for 
each unit type. These values determine the unit’s ability to accomplish particular tasks. 
As above, a unit is any organization that participates in the scenario. Figure 4 shows that 
the unit abilities sub category can be separated into military values and reconstruction 
values. The displayed unit is a health team, and therefore has relatively high values for 


healthcare under the reconstruction values, and relatively low military kinetic values. 


Data and Settings Editor 


Data and Settings 


——- 
SavejLoad Settings | Unit Abilities | stances | Goods Details | Descriptors | Population Agents | Pre-Set Unit Types | Terrain | Display Colours | Combat Modifiers | Outputs | Weather 
fe é tL ad d Bis é a ae: iL nt SLL Bes acon 








‘ Add Unit Remove Output to 
mene Feats eshe) = 


Military Values Reconstruction Values 


Name: |Health Team (Small) Economic Infrastructure 


Domain: Land & | Mobility: [30 








Firepower: | | Change Attitude: lo 
cH —- 
Protection: | | Crime: Qo 


Sensors: |: ] Policing: = 0 


Collateral |p Internal Order 


Intelligence: | 
Damage: 


Administrati 
Social | ministration 
Camoflague: | 
Physical | 
Camoflague: | 


Domain Modifiers 


Manpower: | 


Max Footprint /- 
Size: Ll 





Logistics Cost: {c 


Logistics Provision; f 








Save Changes 








Figure 4. Example of Unit Abilities in PSOM 
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Qi Stances 


Stances provide a catalog of actions that a unit or faction can take toward another 
entity in the game. Stances are broken in categories based on a main stance, and each 
main stance has one or more sub stances. The stances are similar to the operations and 
the sub stances are similar to tactical tasks. For example, a main stance can be attack and 
the sub stances are the multiple types of attacks, such as ambush or indirect fires. This 
category allows the user to add main stances and sub stances and make changes to the 
numeric values that create the particular stance array. A unit’s stance determines the 
intensity of combat tasks, information operations, humanitarian tasks, or logistical 
support. Figure 5 is a screen shot from the stance “attack” with sub stance “ambush- 
direct fire.” So a unit given this stance, regardless of unit type, has the corresponding 
values. Also provided in Figure 4 is a list of the main stances available to all units. 


Special units (e.g., air and sea) have specific stances. 





Data and Settings Main Stances 


SavefLoad Settings | Unt Abilties | Stances | Goods Detais | Descriptors | Population Agents | Pre-Set Unit Types | Terrain | Display Colours | Combat Modifiers | Outputs | Weather| (Coalition Units) 


, Add Main Remove 
Main Stance: |Attack/Enforce be! 


ib Stas — Build Humanitarian Aid 


Effort | Modifiers 





Occupy Infrastructure 
Destroy Infrastructure 


Name: Ambush - Direct Fire 
Clear 


eee 
Indirect Fre/Harassment Hae Attack/Enforce 
Attack Logs Infrastructure: [2 Attack Units: |1 Attack Population; (0 


Destroy Human Capital (Covert) rae 
ack Human 














Destroy Infrastructure (Covert) Attack Logs: [0 
Ambush - IED Capital: 3 aL | ais 
an A a Control/Stabilize 
fen 
Support Fires in Guard — Guard ——— 
Guard Units: [0 Infrastructure: {2 Population; [9 
Renata Transition 
Build Human Build i . |o 
Capital: eo Infrastructure: |° Pe 
Other Miltary Tasks Information Operations 
Train Units: 0 Receive Training: [0 QRF: 0.3 _ 
Recce/Intel {g > . 
Gathering Reconnaissance 
Crime and Policing - - = 
Traffk: [0 Extortion: [0 Counter-Crime: {9 
Add Sub Remove Sub ist] 
other = Logistics/Headquarters 
Logs af lax No Contacts 
Loos Modify Perceptions: |0.2 perManDay: (0-015 
Logs Cost: [1 c2cost: [1 Average Size of [39 Counter-transition 


Force on Stance: 




















Figure 5. Example of Stance Setting in PSOM 
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3. Goods and Services 


This category allows for categorical costs, production details, and population 
expectations of provided goods and services. Goods and services can be provided by 
participating factions as any combination of palliative aid, infrastructure, and human 
capital. Different types of goods can be added to the model, some common examples of 
which are power, sanitation, food, and security. The modeler can assign values such as 
cost per person for palliative aid, maintenance costs, production details, and the 
population expectation for a particular good or service. A population’s expectations 
compared to what is provided effects the population’s consent toward particular factions. 
The effect of a particular good or service provided in PSOM on population consent is 


based on public choice theory (Body, 2008, p. 45). 
4. Terrain 


PSOM can represent different types of terrain. Unit firepower, protection, 


detection ability, and mobility are all modified by terrain settings. 
= Population Agents 


Agents are “Semi Autonomous groups of ‘similar’ people of a defined size. An 
agent may make a number of decisions based on the conditions and environment 
surrounding it” (Parkman, 2008, p. 41). This category is used to create generalizations 
about the overall population. The number of people in a particular agent (population 
resolution), group decision actions (decision radius), and criminality (average time in 


prison for crime) settings are examples of attributes in a population agent. 
6. Combat Modifiers 


These settings allow for generalizations about combative units. These are often 
averages that are used in later calculations. Examples are average distance moved, 


planning delay, average leadership values, and comparative values for conscripts versus 
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veterans, and typical fire control status (ROE). These values provide a sort of base case 
or expected values for military or paramilitary unit types using subjective subject matter 


expert inputs. 
as Outputs 


PSOM provides a wealth of deterministic metrics that can be displayed via the 
graphical user interface (GUI) or sent to a comma separate value file (CSV) for analysis. 
Multiple metrics relating to PMESII can be extracted from the outputs tab. This is 
important because in irregular warfare success is often subjective and there is no clean 
list which provides the conditions for victory. Rather, many metrics are interwoven and 


dependent on one another. There is more discussion on this topic in the results section. 
E. GAME PLAY 


PSOM is designed to be a human-in-the-loop wargame. By definition a wargame 
is a tool for the modeling and exploration of human decisions processes in the content of 
military action (Perla 1990 p. 261). 

An action model that is disconnected from the decision maker’s intuition 

and from the concepts he or she is familiar with does not permit interplay 

between the decision maker and the model. In short, complicated, non- 

intuitive action models require decision makers to accept the implications 

of the model on blind faith. Action models should aid decision makers not 


replace them. (Committee on Organizational Modeling: From Individuals 
to Societies, 2008, p. 321). 


Therefore, the way in which the game is played is critical. The game should be 
intuitive in how it is played so as to ensure the semantics of game play do not interfere 
with player decisions and lessons learned from game play. PSOM is implemented by two 
separate installations, one server and multiple clients. Clients represent the multiple 
factions and units involved in the wargame. Figure 6 was taken from the PSOM 


specification manual and displays this. 
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Client 1 Client 2 


Same Scenario File 
sent to Clients 


Server 


Low Orders files 
Client 3 returned to Server Client 4 


Figure 6. Setup of Game Play [From (PSOM Specification Manual)] 





































































































The client interface shown in Figure 7 is the primary means in which the players 
implement unit orders. Prior to each time step the client updates the client interface. The 
client can determine the unit’s stance and sub-stance toward particular factions, initiate 
movement, loosen or tighten the unit ROE, change the unit’s force protection level, along 
with other faction-specific updates. Once the player (client) has updated this interface for 
all units within the corresponding faction, the player sends his complete order to the 
server. It is possible for a client’s order to be scripted as well. For example, the game 
director can submit a predetermined order for a new insurgent group to attack a target on 


a particular turn. 
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Client Mode: Turn 0 (D -30) Faction - Coalrtion 
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Figure 7. PSOM Player Interface [Best viewed in color] 


Similar to the client interface for factions, the higher level game interface shown 


in Figure 8 is used to update the political and strategic environment the game. 
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Client Mode: Turn 0 (D -30) Faction - Coalition 
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Figure 8. High Level Game Interface 


Once all of the clients/players have sent their updated orders to the server, the next time 
step for the game can be run and the outcomes of the previous actions are provided. At 


this time the game turn process is completed again. 


Time steps for the wargame can be variable; however, a normal time step is thirty 
days. Therefore, all contacts, humanitarian actions, policy changes, training missions, 
population actions, and IW considerations for thirty days will be played out between each 
turn. Faction activities will influence the population, resulting in changes in population 
support, HUMINT, crime, etc. These will all play out during the thirty-day period and 
have effects on the status of the host nation in which the war is being fought at the end of 


the time step. 
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F. OUTPUT 


PSOM can output a tremendous amount of data as output from each turn of the 
game. This is imperative because “No definitive list can be developed due to the 
somewhat subjective nature of success” (Body, 2008, p. 16). Because of the interaction 
between factions, many measures of effectiveness are explained in regards to a 
relationship between factions and the population. This level of detail allows particular 
factions to know how their conduct effects the population and other factions. With 
careful analysis of the multiple outputs a faction can determine the causality of actions by 
subordinate units. However, these relationships are not often apparent with the basic 
outputs displayed between turns and take some experience with PSOM to find. PSOM 
does not provide a list of stochastic possibilities that units could take in regards to faction 


actions. 


In approximate accordance with military doctrine, the primary MOEs for PSOM 
are security and consent (Stability Operations, 2008, pp. 1-33, 1-77). Both of these 
measures relate to the population and are represented on a scale ranging from | to 10. In 
the philosophy documentation (Body H. , 2008), security is defined as the key MOE in 
PSOM. Security tracks the perceived risk of violent death of a population agent in a 
scenario. This MOE is applied only to the population agents in regard to factions. It is an 
aggregate of the security amongst all ethnic groups within that population’s grid square. 
The security MOE is subject to a memory effect in that perception takes time to catch up 
with reality. It is calculated from deaths to an ethnic group within a cell while accounting 
for the size of that ethnic group. Ifa faction actively provides security to a population the 


metric will credit the faction with the security provided. 


The consent MOE depicts the degree to which the population supports and 
submits to the will of a faction (Jon Parkman, 2008). Unlike the security MOE, consent 
is not an aggregate of ethnic groups; rather, it is based on the perception of a particular 
ethnic group toward a faction. So, for example, one ethnic group can have a high consent 
toward the national government while another ethnicity is the opposite. The consent 


MOE is subject to memory effects and it is modified by the ideological differences 
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represented by the Nolan Chart. The value of consent in regard to each faction is a 
function of the level of goods and services provided, the marginal gain coefficients, and 
the population agents’ expectations. This approach is supported by the standard rational 
choice theory. “The basic idea behind rational choice theory is that people do their best 
under prevailing circumstances” (Green, 2002, p. 5). Although these are the primary 


outputs, Table 2 shows the PMESII indicators and their equivalent PSOM output. 
































PMESII Indicator Corresponding PSOM Output 

Political Legitimacy Population Consent toward own Government, Security, 
Rule of Law, Corruption, Provision of Essential Services 

Military Casualties, Contacts 

Economic Production, Reconstruction, Income, Human Capital, 
Growth Rate 

Social Fear, Rule of Law, Economic Factors 

Information HUMINT, Headlines 

Infrastructure Infrastructure, Economic Production, Human Capital 





Table 2. PMESII Indicators in Relation to PSOM Output 


PSOM also provides the user a headlines function which is intended to highlight 
activities that would most likely become open source press releases. Such topics as 


civilian deaths, enemy deaths, and terrorist attacks are general headlines. 


The taxonomy of the parameters and the responses for the Peace Support 
Operations Model provides a logical and intuitive environment to simulate contemporary 
warfare. As explained earlier in this chapter, the inputs required to design and execute a 
simulation in PSOM cover a vast space from combat capabilities to social behaviors. 
This study is able to take advantage of this design to create an experiment that will 
evaluate the model based on its underlying categorical blueprint. By initially completing 
multiple designs of experiments based on the model’s categorical setup, we can then 


determine and aggregate the influential factors to further explore the model in its entirety. 
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Hil. SCENARIO DESIGN 


One of the most important jobs of wargame developers is to assess the 
validity of the game’s results and processes in light of the real world. 


(Perla, 1990) 


A. INTRODUCTION 


In evaluating military simulation models it is critical to use a realistic and 
accurate scenario that is pertinent to current military operations. This chapter describes 
the macro level military, political, and societal scenario used for all experimentation. The 
chapter then describes the focus areas of the experimentation with more resolution. This 
includes a description of factions, coalition military units, and ethnic groups that are 
critical in the analysis. The chapter concludes with a summary of the measures of 


effectiveness, units employed, and stances in use. 


B. SCENARIO CHOICE 


1. Overview 


Because of the considerable complexity of a campaign level wargame that 
portrays the importance of the population, the first priority is using a scenario that is 
based on thorough research and factual data. Secondly, the scenario should describe a 
military situation that is relatively well understood, thus allowing intuitive and doctrinal 
analysis. Therefore, we used a scenario developed by DSTL based on Iraq in the time 
frame of 2004 onward. DSTL developed this scenario to test PSOM and has conducted 
multiple wargames and considerable adjustments to ensure scenario accuracy. Also, 
PSOM is a fairly new model with minimal documentation and thousands of possible 
inputs; therefore, for this study, we felt it best to use a scenario designed by developers 


who actually understand the simulation. 
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Although the scenario encompasses all of Iraq in the given time frame, in order to 
allow focused quantitative analysis this study primarily concentrates on the coalition 
forces, the Sunni Nationalists, and the Sunni Ethnic Group Population Agents. All 
factions and ethnic groups are discussed in this section; however, the emphasis is on the 


aforementioned groups. 
2. The Macro Level Scenario 


As previously stated, the scenario used is Iraq 2004 onward. The force structure 
of coalition forces closely matches that of mid 2004 to 2005. The enemy force structure 
is loosely based on known information, and the population’s attributes are also reflective 
of this geo-spatial information. Table 3 describes the factions in play and their respective 


sub-factions. For the majority of simulation runs, twelve 30-day time steps, equivalent to 





one year of combat, are played. Therefore, the game is a terminating scenario. 





Coalition 


Office of the Matyr 
Sadr 


Iraqi Governemnt 


Sunni Nationalists 


UN, IGO’s 





Combined Air 
Operations Center 


Jaish al-Mahdi 
Commander 


New Iraqi Army 


Ba’ athist 


United Nations 
Assistance Mission 
for Iraq 





Combined Joint 
Special Operations 
Task Force 


Police Service 


Al-Qaeda in Iraq 





Multi National 
Division (B) 


Facility Protection 
Service 


Tribal Militias 





Multi Nation 
Division (NC) 


Development Center 





Multi Nation 
Division (CS) 


Shi’a Militias 








Multi National 
Division (SE) 


National Police 














Multi National 
Division (W) 





Coalition 
Provisional 
Authority 





Multi National 
Forces Iraq Corps 
Assets 

















Table 3. 
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Breakdown of Iraq Scenario Factions 





The population consists of three primary ethnic groups (using the PSOM 
definition of ethnic group): Sunni, Shi’a, and Kurds. During Saddam Hussein’s reign 
these ethnic groups separated themselves across the country, creating geo-ethnic areas 
throughout the entire nation. For this particular scenario the coalition defined areas of 
responsibility closely align with these ethnic divides. This is shown below in Figure 9. 
Due to the sheer size and complexity of this scenario, we have focused our emphasis on 
Sunni civilians, the Sunni Nationalist factions, and the coalition forces that are assigned 
to the areas in which Sunni’s and the Sunni Nationalists live and operate. Figure 9 
displays the Office of the Martyr Sadre Faction (JAM), the Sunni Nationalist Faction, and 


the population of ethnic groups by region at simulation time step zero. 
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Figure 9. Break Down of Population, JAM, and Sunni Nationalists by AOR [Best 
viewed in color] 
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Figure 10 shows the Map of Iraq broken down by Area of Responsibility used for 
this study. 
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Figure 10. Coalition Force Structure by Area of Responsibility [Best viewed in color] 





Cc. SITUATION 


By late 2004, coalition forces were well established with over 150,000 troops in 
Iraq. Although the new Iraqi government has control over the country, the population’s 
support is questionable across the nation. The growth of Al Qaeda in Iraq (AQI) has 
been noted throughout the country and increased violence between ethnic groups has 
many feeling a civil war is inevitable. At the same time, a tremendous emphasis has been 
put forth by the coalition to build Iraqi Security Forces and the Iraqi Army is starting to 
take shape. The country’s infrastructure is still weak, proven by a lack of continuous 


power and potable water throughout the country. 
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1. Factions 


a. Coalition Forces 


The faction of coalition forces consists of over sixty combat maneuver 
battalions and the required combat support and service support. These forces are spread 
throughout the country with the majority focused on Baghdad and MND(NC). In 
addition to the combat units, numerous air wings, indigenous force training teams, and 
reconstruction teams support the coalition mission. The coalition’s primary stances 
during this phase of the war are to provide security and assist in the rebuilding of the war- 


torn nation. 
b. Sunni Nationalists 


The faction of Sunni Nationalists consists of the emersion of Al-Qaida in 
Iraq (AQJD) and tribal militias. AQI consists of ten cells each with approximately forty 
members. Their primary focus is either attacking the coalition forces or the Iraqi 
government through the destruction of infrastructure. As can be seen in Figure 9, the 
majority of these forces are in Bagdad or the surrounding areas. The majority of the 
Sunni Nationalists are the nine tribal militias which consist of approximately 700 troops 
each. These units are primarily anti-coalition and anti-Iraqi government as well. The 
Sunni Nationalists also maintain a small number of training and reconstruction teams 


which operate in the central to northern portions of the country. 
Cc. Office of the Martyr Sadr 


The faction of the Office of the Martyr Sadr (JAM) consists of fourteen 
militia battalions, but for purposes of this scenario only seven are activated and seven are 
in hiding. In addition, JAM has seven training units and seven construction units to help 
win populace consent. As can be seen in Figure 8, the only area where JAM and Sunni 
Nationalists both operate is in Baghdad; the remaining JAM units focus primarily in 
MND(CS) and MNS(SE). JAM?’s primary stance in this game is to protect the Shi’a 
population from both the Sunni Nationalists and the coalition forces. 
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d. Tragi Government 


The faction representing the Iraqi government primarily consists of 
security forces. The Iraqi Security Forces are comprised of forty police Battalions, forty 
facility protection service (FPS) groups, five construction teams, five training teams, 
twelve Militia Battalions, twenty-seven national police battalions, and some additional 
recruiting and information operations units. These units are primarily focused on the 


security of the nation. 


e. United Nations and Inter-governmental Organizations (UN, 
IGO) 


The UN and IGOs faction consists of thirty-three humanitarian aid teams 
spread throughout the nation. Their primary role is to provide aid in the form of medical 


assistance, education, subsistence, and infrastructure to the population. 


2. Population 


a. Sunni Ethnic Group 


In this scenario the Sunni population comprises approximately 25% of 
Iraq’s total population (see Figure 8). As a group, the Sunni do not approve of the 
coalition forces and have only a slightly higher opinion of the current Iraqi government 
than of the coalition forces. However, the Sunni population has very strong consent for 
the Sunni Nationalists and their cause. The Sunni population holds the Office of the 
Martyr Sadr in contempt and fears JAM and coalition forces equally. The population 
primarily lives in the north central areas of Iraq and except for in Baghdad is segregated 


from the Shi’a populace. 
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b. Shi’a Ethnic Group 


In the given scenario the Iraqi population is 48% Shi’a. As a group, the 
Shi’a accept coalition forces with consent equivalent to the Office of Martyr Sadr. The 
Shi’a generally support the current Iraqi government, but condemn the Sunni 


Nationalists. The Shi’a primarily reside in Baghdad and southern Iraq. 
c. Kurd Ethnic Group 


The Kurd ethnic group provides for the remaining 25% of the Iraqi 
population. They primarily live in the most northern areas of Iraq and support both the 
coalition forces and the Iraqi government. The Kurds are impartial toward the Office of 


the Martyr Sadr, but hold the Sunni Nationalists in contempt. 


3. Game Execution 


a. Overview 


Normally, the intent of the coalition faction in this wargame is to actually 
play the PSOM Iraq scenario with the goal of creating an independent, secure Iraq with 
lucrative economic conditions and a legitimate government fully supported by the 
population. However, this is not the case for this experiment. Rather, our goal is to test 
the realm of possible outcomes received by the users of PSOM. For this we focus on the 
security levels and consent levels throughout the country as we change faction activities 


and scenario parameters. 


The majority of simulations executed represent one year of combat. 
Starting conditions were not changed within simulation runs, so if a faction begins the 
year with an aggressive stance and a relaxed set of ROE, these characteristics are 
maintained for the entire year. This is an identified limit of the study. The human-in-the- 
loop technique allows for factions to adapt to reactions of the population and other 
factions. Unfortunately, in using the pre-scripted batch technique to run the scenario we 
cannot adjust the factions’ actions throughout the year. This also pertains to unit 


movement. Therefore, a unit could not move out of its initial allotted footprint. This 
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being said, because each time step represents thirty days of combat, results of the 
previous month are carried over to the current month’s fight. For example, civilian and 
military causalities, population reaction, and infrastructure growth all are determined time 


step by time step throughout the game. 
b. Measures of Effectiveness 


The primary measures of effectiveness addressed are the consent of the 
Sunni population toward the coalition forces, the consent of the Sunni population toward 
the Iraqi government, and the security level throughout the country. PSOM displays 
these outputs in 50km X 50km grid squares (see Figure 11). This system results in 135 
separate responses as the game is played. To allow for analysis, the responses are 
aggregated by coalition Area of Responsibility and the entire country. This allows for 
regional averages for the MOEs, and therefore reduces 135 outputs for three metrics to 
seven outputs for the three metrics. However, these metrics are based on geo-spatial 


representation and not the quantity of people within the cell. 





Figure 11. The Graphical Display of Ethnic Groups Consent for the Coalition from 
the Peace Support Operations Model [Best viewed in color] 
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Cc. Units Played 


Due to the nature of a campaign level model, the number of maneuverable 
units can overwhelm analysis, causing excessive noise and confounding. To prevent this, 
the study focuses on varying the attributes and actions of selected units and population 
agents. A total of thirty-seven coalition maneuver battalions which are located in 
MND(B), MND(NC), and MND(W) are played, seventy-five Sunni Nationalist units are 
played, forty-seven UN IGOs are played, and 535 Sunni population agents, which 


represent 5.35 million Sunni civilians, are the population focus. 
d. Unit Stances 


The complete taxonomy of stances and sub stances in this scenario ranges 
the possible set of doctrinally defined operational and tactical tasks. For this study, 
coalition forces are limited to the following stances: providing aid, securing, attacking, 
and withdrawn. Sunni Nationalists are either attacking coalition forces, attacking Iraqi 
infrastructure, or withdrawn. IGO’s are either providing aid or withdrawn. The stances 
are uniform across the units mentioned in the previous paragraph. So, if a particular 
design point includes coalition stances with a stance of “withdrawn,” all thirty-seven 
coalition maneuver battalions are withdrawn. This is a noted limitation to this particular 


study and should be examined further in a follow-on study. 
D. SUMMARY 


The contemporary Iraq scenario used for this study was developed and thoroughly 
evaluated and updated by the Defense Science Technology Lab UK, who also developed 
the Peace Support Operation Model. It is the most robust scenario developed for PSOM 
and the best for this study. This is of significant importance due to the tremendous 
amount of social parameters that a PMESII model must account for. Due to the number 
of parameters and units the study narrows the scope of concern to primarily the Sunni 
population, the Sunni Nationalist Faction, and the coalition maneuver battalions operating 
in the areas where the Sunni population exists. By narrowing this area of focus the study 
can use intelligent Design of Experiment in conjunction with modern computing power 


and basic data mining to analyze the PSOM’s output. 
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IV. EXPERIMENTAL DESIGN 


Everything should be made as simple as possible, but not simpler. 


Albert Einstein 1933 


A. INTRODUCTION 


In the simplest form the purpose of this research is to gain a basic understanding 
of the Peace Support Operation Model. In particular, we need to determine which factors 
are significant in the model and how they affect the responses (Kelton, 1991, p. 657). In 
order to quantifiably analyze PSOM, this study follows the data farming process 
described by Horne (2004). Our implemented iterative data farming process entails a 
four step procedure: Define factors of interest, create a design of experiment (DOE), run 
the simulation experiments in parallel on a computing cluster, and conduct data mining. 
The initial definition of interesting factors is a research and subject matter centric step to 
begin the process. The use of DOE allows us to gain “detailed insight into the model’s 
behavior” (Kleijnen, 2005, p. 266), control the bias and confounding of factors, and 
explore a tremendous number of possible parameter combinations efficiently. Data 
mining is then used to analyze responses from the created vast parameter space (Phillip 
Barry & Koehler, 2004). This is essential as a low-resolution model which encompasses 
social implications has no closed form solution. We can then take this large data set and 
develop relatively simple formulas that act as a proxy for the actual simulation (Kelton, 
1991). These “meta models” can then be used to determine which factors are significant 


within PSOM and what effects their manipulation might have. 


This chapter describes in detail the variables selected as factors in the experiment, 
the methodology used to create the DOEs, and the tools created at the Naval Post- 
graduate School to take the PSOM wargame and convert it into a data farmable 


simulation. Next, the design of experiments processes used are discussed, providing the 
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reader insight into how the study efficiently tests a large space of possible combinations 
of input parameters (Horne, 2004). We then discuss the meta modeling techniques used 


to data mine the tremendous amount of output collected from the experiments. 


B. FACTOR SELECTION AND METHODOLOGY 


1. Overview 


In analyzing the variables and comparing results, factor selection proves vital to 
an efficient and worthwhile study. PSOM is a complex model and the number of factors 
used to build the Iraq scenario is too immense to study in its entirety. For example, there 
are twelve factors used to describe each particular unit and there are an additional thirty- 
three attributes used to create each unit type. For example, A/1—26 IN has twelve specific 
settings and every generic infantry company has thirty-three attributes in common with 
all infantry companies. Therefore, any one of the 200 coalition units has forty-five 
variables which describe its totality. In addition to the number of factors, there is no 
published PSOM user’s manual (there is a draft manual cited throughout this work), and 
many of the variables are not constrained to particular numeric ranges. Naively selecting 
the factors without regard to their appropriate ranges would result in a design where 
important factors are washed out by noise or dominated by unknown limits on 
parameters. This section explains the tactics used and factors selected in building the 


experiment. 


As previously stated, the scope of the study is limited by the focus on 
geographical areas inhabited by the Sunni. To further narrow the scope, we looked to the 
opinion of subject matter experts on irregular warfare, combat modeling, PSOM, and the 
DoD’s needs in IW modeling for the first stages of the experimental design. Although 
this reduces the number of factors tremendously, multiple designs were needed to create 
practical experiments. These experiments are categorized by scenario design and settings 
design. Finally, because this is an iterative process, the emersion of results, analysis, and 


expert opinion creates a cumulative design of experiment. 
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2. Scenario Specific Design of Experiment 


The scenario specific design of experiment analyzes the parameters found in the 
scenario file. These factors are either set during scenario development or changed by 
players during the game. Many of these factors are crucial assumptions made about 
either population agents or factions. Of note, not all factors are quantitative in nature. 
For example, a unit’s stance is an action from a list of possible tactical or operational 
tasks. These categorical factors increase the number of design points tremendously and 
are therefore limited in scope. Table 4 shows the list of scenario design categorical 
factors, their experimental ranges, and brief descriptions. Stance changes affect all units 
in the area of game play at the beginning of the game and they maintain these stances for 


the entire 12-month period. 






































Factor Level 1 Level 2 Level 3 Level 4 Description 
n Combat Units CBT Units CBT Units CBT Units 37 Coalition 
Coalition i ; Bee : 
Sides Attacking Securing Providing Withdrawn maneuver 
Sunni Humanitarian Aid BN’s in Sunni 
Nationalists— inhabited 
Sub Stance Build regions take 
Clear Patrol Infrastructure this stance 
Sani Attacking U.S. | Attacking Iraqi | Withdrawn 75 Sunni 
Nationalist Sovemment Nationalist 
Stance Units (AQI 
and Militia) 
in Sunni 
Destroy inhabited 
Sub Stance Ambush Infrastructure cs take 
this stance 
1Gorsimees ven. eee te 47 IGOs 
umanitarian 
Aid throughout 
Iraq take this 
stance 
Coalition Yes No Determines if 
Shares the coalition 
Intelligence and Sunni 
with Sunni Nationalist 
Nationalist share 
information 
Table 4. | Categorical Factors used in the Scenario DOE 
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Table 5 is a list of all the quantitative factors in the scenario DOE, their 


experimental range, and a short description. 















































Factor Experimental Description (Parkman, 2008) 
Range 

Coalition 1 (Loose) and 5 (Tight) representing the degree to which the unit is willing to 

ROE Level 1-5 cause civilian casualties in order to complete its task. 

Coalition An integer between 1 (Low) and 5 (High) representing the degree to which the 

Force I=) unit is willing to suffer its own casualties in order to complete its tasks. 

Protection 

Level 

Sunni ROE 

evel 1-5 See above. 

Sunni Force 

Protection 1=5 See above. 

Level 

Sunni me ; ; : ; 

Politcal 0-100 This is a value between 0 and 100, which give the Faction’s ideology. based 

ideology on its views on personal freedom, through the Nolan chart system, as shown 
below. 

Sunni 

Marginal 0.30.6 These values, one for each Good Type, control the level of importance that the 

Gans group places on the provision of that good type. 

Sunni : . : 

Marginal 0.30.6 This value controls the level of importance that the ethnic group places on 

Gains Security. 

Security 

Sunni Initial ae . 

Consent 2-8 These values set the initial levels of Consent towards each Faction that are 

Coalition possessed by the generated Agents at the start of play. 

a a 2-8 This value sets the initial level of threat that a population agent feels toward 

toward each faction. 

coalition 

Coalition : ; bo 

Casualty 0-100 Casualty tolerance value, which controls how many casualties the unit will 

ilersnice bear each turn before the deterrence function begins to have an effect. 

ete 0-100 The level of competence in the leadership of the unit. 

Coalition : as . ; ; ; 

Experience | 0-100 The level to which the Unit iS trained and experienced in conducting 
operations in a PSO type situation. 

Coalition ; . : fs we 

Reputation 0-100 The degree to which the population perceives that the unit is unwilling to 
conduct offensive operations against them. 

Coalition : : ee : : 

Turaeat 0-12 The values for calculating the unit’s familiarity with the local environment. 

Location 

















Table 5. | Continuous Factors used in the Scenario DOE [From (Jon Parkman, 2008)] 
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3s Explanation of Factors 


a. Stance 


The stance that a faction takes determines the actions of that faction. 
Therefore, the choice of stance is critical. This experimental design emphasizes the 
different initial stances and the resulting responses. If the players’ actions do not provide 
reasonable reactions from the game, then its use is questionable. We expected to see 
considerable interaction between the coalition and Sunni Nationalist stances. This thesis 
looks into the stances of the coalition forces, Sunni Nationalists, and the IGOs. These 


factors are normally controlled by the players. 
b. Rules of Engagement 


The players also set each unit’s ROE. This provides the player the ability 
to designate the degree in which each unit is willing to cause civilian casualties in order 
to accomplish its mission. A setting of 1 is a very loose ROE and would be indicative of 
a VBIED detonated in a civilian populated area, where as an ROE of 5 represents a sniper 


who would only engage upon positive identification of an enemy. 
c. Force Protection 


Force protection is similar to ROE in that the players control this setting 
for each unit on a scale from 1—5. A unit’s force protection describes its willingness to 
assume risk to accomplish its mission. A value of 1 is low risk and a value of 5 is high 


risk. 
d. Political Ideology 


Political ideology is a population agent attribute which falls on a scale of 
1-100 (see Chapter H]). This ideology is important in the development of population 
agents and therefore should play a role in the population’s consent toward factions. 


Ideologies are defined in the scenario setup. 


39 


e. Marginal Gains for Goods 


Marginal gains determine the importance of a particular good for each 
ethnic group. Marginal gains are on a scale of 0—1 and are a primary characteristic in the 
building of population agents during scenario development. Because these play a key 
role in the population’s development and rules it is essential to understand their impact on 
the game. During the initial iterations this factor proved very powerful to game 


outcomes, so the experimental range was limited to 0.3—0.6. 
f Marginal Gains for Security 


The security marginal game is similar to that for goods. In this study its 
value was changed separately because a society’s concern for security is more likely to be 
influenced based on combat actions. Thus, the consent corresponding to marginal gain 
security is changed based on security provided as opposed to goods production. During 
the initial iterations this factor proved very powerful to the game outcome so the 


experimental range was limited to 0.3-0.6. 
g. Initial Consent 


The initial consent an ethnic group feels toward a faction is a tremendous 
assumption in scenario development. As with consent, this factor is on a scale from 0 to 
10. With any scenario it is crucial to understand how initial conditions can affect 


simulation execution. 
h. Initial Threat 


The initial assumption of threat an ethnic group feels toward a faction is 


similar to initial consent, and its implications should be explored. 
i. Coalition Casualty Tolerance 


Coalition casualty tolerance controls how many casualties the unit will 


bear each turn before the deterrence function begins to have an effect. This value is on a 
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scale of 1 to 100 and is used to describe each unit during scenario development. Often in 


irregular warfare numerically small levels of casualties can create strategic implications. 
J: Coalition Leadership 


Coalition leadership describes the level of competence in the leadership of 
a particular unit. This value is on a scale of 1 to 100 and is used to describe each unit 
during scenario development. The United States relies heavily on leadership, making this 


factor an essential assumption. 
k. Coalition Reputation 


Coalition reputation describes the perception the population has on a 
particular unit. This value is on a scale of 1 to 100 and is initialized during scenario 
development. In warfare, dealing with the populace the reputation of a unit should have a 


direct effect on the collection of HUMINT, coalition freedom of movement, and trust. 
L. Turns at Location 


Coalition turns at location describes the number of time steps which a unit 
has been in a particular location and thus how familiar it is with that area. This value is 
on a scale of 0 to 100 and is initialized during scenario development. This factor has 


implications in the strategic emplacement of units and deployment timelines. 
4. Setting Specific Design of Experiment 


The settings design of experiments focuses completely on the settings factors in 
the Peace Support Operation Model. These factors are primarily generic in nature in that 
they are descriptions of units, populations, and conditions by type rather than tied to the 
particular scenario. The settings describe broad unit abilities, dictate stance attributes, 
and create values used to modify mathematical functions within the simulation. For 
example, these parameters can be used to provide insight on increasing unit manpower or 
mobility, giving combat units the ability to provide humanitarian aid, make combat more 
aggressive in nature, or determine if a patrolling unit acts to protect civilians or gain 


intelligence. 
4] 


Because of the large number of these factors (sixty-six) and their similar nature, it 
is best to describe the purpose of their exploration upfront and give only a brief 
description of each. First off, it is crucial to understand the parameters used in any 
model prior to using it as a decision aid or training tool. Many of these parameters are 
not well defined in any literature, nor do they have limits to their values. Using an 
intelligent DOE we can explore the space of these factors and determine which, if any, 
will have a profound effect on the simulations outcome. This can only lead to better use 
of the model as a training wargame and analytic tool. Secondly, many of these attributes 
are quantitative descriptions of unit types. Often military comparative studies are used to 
modify unit attributes in determining fielding and acquisition questions. If PSOM can 
provide such comparative analysis in the world of irregular warfare it can prove a useful 
simulation. In order to mitigate any one factor from dominating the experiment, these 
factors are varied +20% from the original values in the scenario developed by DSTL. 


Table 6 lists all the factors used in the settings DOE. 
























































Factor Experimental | Description 

Range 
Unit abilities set attributes of the particular unit types. These were cast across maneuver companies. 
Unit Fire Power 64-96 Level of firepower per man 
Unit Protection 80-120 Level of armor per man 
Unit Sensor 40-60 Level of sensors per man 
Unit Intelligence 16—24 The ability per man to recognise and classify 

Level to which the element can blend in 

Unit Physical Camouflage 0.8—1.2 surroundings 
Unit Social Camouflage 0.8—1.2 As above but in social population 
Unit Mobility 24-36 Average speed in km/h the element moves 
Unit Change Attitude Ability 2.4-3.6 Ability to change the attitude of the population 
Unit Crime 1.6—2.4 Ability per man in the element to commit crimes 
Unit Policing 2.4—-3.6 The ability per man in the element to counter crime 
Unit Collateral Damage 5.6—8.4 The level of expected collateral damage by unit 
Unit Size 72-108 Number of men in the unit 
Unit Palliative Aid Ability 0.24—0.36 Unit’s ability to provide palliative aid 











Stance settings describe the attributes of a particular stance. These were applied to stance Secure/Patrol. 
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Factor Experimental | Description 
Range 

Stance Attacks Unit 0.24-0.36 Proportion of time in the turn spent attacking 
Stance Protect Population 0.32-0.48 Proportion of time in the turn spent protecting 
Stance Provide Aid 0.16-0.24 Proportion of time in turn providing aid 

The level to which the population will be reassured 
Stance Modify Perception 0.32-0.48 or intimidated by the Unit’s actions 
Stance Extort 0-0.3 Proportion of time which a unit extorts 
Stance Counter Crime 0.24—0.36 Proportion of time which a unit counters crime 
Stance Intel Gather 0.4-0.6 Level which a unit’s actions allow it to gather intel 
Stance QRF 0.32-0.48 Proportion of time which a unit is on QRF 
Stance Average Size 24-36 Average size of unit on stance 
Stance Protection Modifier 0.64—0.96 Modifier to units protection value while on stance 
Stance Mobility Modifier 0.8-1.2 Modifier to units mobility value while on stance 
Stance Detectability Modifier 0.96-1.44 Modifier to units detectability value while on stance 
Stance Detection Modifier 1.2-1.8 Modifier to units detection value while on stance 
Stance Recognition Modifier 0.8—1.2 Modifier to units recognition value while on stance 





Settings that control the generation of the populatio 


n. These are applied to the entire population 





Distance in km that an agent will look across to find 























Population Decision Radius 40-60 a better location 

Population Memory Coefficient | 2.4-3.6 Half life (in turns) of the current consent value 

Population Consent Political Controls the effect of ideology differences on 

MScaler -0.64—-0.96 Consent (gradient) 

Population Consent Political Controls the effect of ideology differences on 

CScaler 0.08-0.12 Consent (intercept) 

Population Average Term In : , 

Pro. 24-36 Average length (in turns) of a custodial sentence 
% of a population that will self-present at a 

Population Self Presenters 0.4—0.6 hospital—outbreak 

Population Police Clear Rate 0.2-0.3 Gradient of the police clear rate 

Population Infection MargGains_ | 0.16-0.24 Related to the infection variable 





The following are time variables used in the PSOM 


queuing algorithm 

















Combat Mod Que Decay Rate 0.012-0.018 Intelligence decay rate 
Percent Force on Duty 0.24-0.36 Proportion of a military unit on duty at any one time 
Length of time in hours that it takes to plan 
Planning Delay 4.8-7.2 operation 
Length of time in hours that it takes to carry out a 
Operation Time 3.2-4.8 small operation 
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Factor Experimental | Description 
Range 

Length of time in hours the portion of a unit in 
Recuperation Time 8—12 contact should be unavailable 

Average distance that a force will have to move 
Avg Distance Traveled 6.49.6 within a square to prosecute a contact 
Max Fatigue 0.64-0.96 Unused 
Fatigue Drop Off Factor 3.24.8 Unused 
Force Protection Mean 2.43.6 The mean of the force protection scale 
Force Protection K Value 0.24-0.36 Calibration factor 

Level to which force protection actually impacts 
Force Protection Mod 0.24-0.36 casualties 
Mean ROE 2.4-3.6 Mean of the ROE scale 
ROE K Value 0.24-0.36 Calibration factor 

Level to which rules of engagement actually impact 
ROE Mod 0.24-0.36 civilian casualties 
Max Leadership Mod 1.6-2.4 Maximum modifier possible with oo leadership 
Min Leadership Mod 0.16-0.24 Minimum modifier possible with no leadership 

Value controlling the rate at which the value of the 
Leadership Drop Off Factor 0.016—0.024 Leadership modifier curves 

Modifier on the performance gathering of a person 
Familiarization Stranger 0.40.6 who is unfamiliar with his surroundings 

Modifier on the performance gathering of a person 
Familiarization Native 1.6-2.4 who is familiar with his surroundings 
Familiarization Learning 0.08—0.12 Control on the curve between the above factors 
Experience Conscript 0.16—-0.24 Modifier on the performance of a new recruit 

Modifier on the performance of an experienced 
Experience Vet 1.6—2.4 troop 

vane Control on the curve between the above factor. 

Experience Learning Factor 0.1068 ee ee ee oe Brae 

Level of casualties caused to the instigator of a 
Inter Unit Base Casualties ATT | 0.08-0.12 contact 

Level of casualties caused to the defender of a 
Inter Unit Base Casualties DEF | 0.08-0.12 contact 

Baseline combined size of the attacking and 
Inter Unit Base Contact Size 24-36 defending forces in contact 





The following factors are relevant 


to each particular good. 














Goods Expected 0.4-0.6 Population requirement for goods 
Good Protection Value Power 8000-12000 Protection score of power 
Good Protection Value Water 56-84 Protection score of water 
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Factor Experimental | Description 
Range 
Soe Erocuon value Protection score of education 
Education 56-84 
Good Protection Value ; 
HealthCare 120-180 Protection score of healthcare 





Table 6. Factors used for the Settings DOE [From (Parkman, 2008)] 


=H Cumulative Design of Experiments 


The cumulative experiment is the aggregate of results from the previous 
experiments’ analysis and a desire to explore new factors (see Chapter V). After 
completing the very first test run the concept of time step and time for the model came 
into question. We therefore introduced both time step increment and overall simulated 
time as factors on an individual experiment. The scenario DOE brought significant 
insight into both stances and marginal gains and the coalitions limited ability to increase 
consent. We therefore focus on the coalition stances in this design. In regards to 
marginal gains, the marginal gain values’ ranges are very limited in this DOE, and are 
lowered to allow for a greater change in consent. Finally, after analysis of the settings 
DOE, further analysis of the unit capabilities was required. The unit attribute values are 
varied over a greater range in this experiment than the limited 20% deviation we used 


earlier. Table 7 explains the factors used in this DOE. 





























FACTOR VALUES and RANGES 

Coalition Stance Humanitarian Aid or Secure by Patrol 
Time Step 7, 30, 60 in days 

Sunni MG Security 0.1—-0.2 

Sunni Marginal Gains 0.1-0.2 

Unit Fire Power 50-150 

Unit Protection 50-150 

Unit Sensors 50-150 
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FACTOR VALUES and RANGES 





Unit Size 





Unit Change Attitude Per Man 





20-50 











Unit Intelligence Ability 
Table 7. Factors used for the Cumulative DOE 





C. EXPERIMENTAL DEVELOPMENT AND TOOLS 


1. Software Upgrades used to Expand PSOM 


It is important to understand that PSOM is a wargame. It was developed to 
explore irregular warfare with human interaction. Unfortunately, this setup does not 
allow for quick turnover analysis. Adjusting thousands of parameters by hand and then 
executing twelve months of combat would lead to potential human error and an 
unfathomable amount of time. In 2008, when the Naval Postgraduate School first 
received PSOM 2.2.3, the batch mode was not flexible enough for large scale 
experimentation, and there was no way to interface PSOM with the current host data 
farming tools. Such an experiment had never been done. SEED center research associate 
Adam Larson created the tools needed to allow parallel runs of the simulation on a cluster 
of computers. These tools include a command line batch mode for PSOM allowing the 
simulation to run based on predetermined settings completely independent of the 
graphical user interface and a script converter which converts the scenario and settings 
data files to an XML format compatible with current data farming tools. Additional 
changes include the addition of direct control over the simulation’s random seed allowing 
the reproduction of any particular run. Also, in order to allow for data extraction, we 
developed a PSOM post processor that pulls the non-normalized data for each individual 
run, transforms the data to the PSOM scale of 1 to 10, takes the mean of multiple runs if 


used, and determines the variance across the regions. 
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2. Designs of Experiment Used 


A tremendous amount of insight can be obtained in a very efficient manner with a 
well designed experiment. In this thesis, as is often the case with design of experiment, 
the experiment is being used to primarily determine which factors are truly important 
(Sanchez, 2008). To ensure a comprehensive analysis, the experimental design should 
look at a widely ranging combination of factors while at the same time trying to avoid 
confounding to ensure causality can be explained. It is important for the reader to 
understand which design techniques were used throughout this study. However, these 
techniques are tools for analysis and not a subject of this thesis; therefore, they are only 


briefly explained in this chapter. 


For the exploration of the scenario file, this thesis uses three techniques to 
construct an intelligent design of experiments. To account for the categorical variables 


we used an m* factorial design. This design consists of: 


4(Coalition Stances) x 3(Sunni Nationalist Stances) x 2(IGO Stances) x 2(Coalition to Sunni Relations)=48 design points 


A nearly orthogonal Latin hypercube (NOLH) DOE is used for the remaining 
fourteen factors. A design is nearly orthogonal if the maximum absolute pairwise 
correlation between any two input columns is less than .05. Latin hypercubes (LHs) 
provide a flexible way of constructing efficient designs for multiple quantitative factors 
(McKay et al. 1979). However, due to an inherent randomness in their construction, they 
can exhibit substantial correlations among the input variables—thus, inhibiting many 
statistical procedures with which we would like to use to analyze relationships between 
input and output variables. While specially constructed orthogonal LHs exist (see, for 
example, Ye 1998), they often have poor space-filling properties. A design with good 
space-filling properties is one in which the design points are scattered throughout the 


experimental region. 


To address the dearth of space-filling orthogonal LHs, Cioppa (2002) used a 
computationally intense heuristic algorithm to generate and catalogue a set of NOLHs 
with good space-filling properties. These flexible designs allow for the efficient 


examination of many factors and their complex relationships with very low 
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correlations—thereby facilitating powerful statistical analysis (Cioppa & Lucas, 2007). 

For example, by using Professor Susan Sanchez’s NOLH spreadsheet implementation of 
Cioppa’s designs (see http://harvest.nps.edu for the spreadsheet), a DOE with sixty-five 
design points is sufficient to analyze the previously mentioned fourteen continuous 
factors throughout their entire ranges. A basic 2" factorial design requires 16,384 design 
points and would only explore the extreme points of the factors. Finally, by using a 
Cartesian join to create a cross design of the full factorial design with the NOLH design, 
a DOE with 3,120 design points covers a plethora of possible combinations of the factors 


between the two designs, greatly reducing computing expense. 


Each design point (also called an excursion) represents an individual run of the 
experiment at the distinctive parameter values. As discussed in the following chapters we 
found little variance in PSOM output; however, because of the slight stochastic nature of 
PSOM, each excursion was run five times with a different random number seed to 
account for variation due to randomness in the simulation for a total of 15,600 runs on the 
NPS SEED Center’s cluster of fourteen processors. A twelve time step excursion takes 
approximately one minute to run on a standard computer. This process took 


approximately fifteen hours of computing time or about 200 hours of processor time. 


The settings design of experiment was driven by three separate characteristics of 
the parameters. First, all data was continuous in nature, so the NOLH seemed ideally 
suited. However, the NOLH’s described by Cioppa (2002) were limited to twenty-nine 
factors and we wanted to explore sixty-six. Fortunately, COL Alejandro Hernandez had 
developed a tool for his PhD dissertation, Breaking Barriers to Design Dimensions in 
Nearly Orthogonal Latin Hypercubes (Hernandez, 2008), which extends the limits of the 
NOLH far past the sixty-six we required. Finally, many of the settings values are not 
bounded nor well described in the PSOM draft documentation. Therefore, we 
manipulated original settings factors used by DSTL by + 20% to mitigate any parameter 
dominance. The result is a design of experiment with 1,000 distinct design points 


thoroughly covering the ranges of the parameters. Each excursion was run ten times (by 
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now we had seen PSOM ran rather quickly) for a total of 10,000 scenario runs. This 
process took approximately twenty-four hours to complete on the fourteen processor 


cluster. 


The cumulative design of experiment consists of only ten factors, three of which 
are categorical while the remaining seven are continuous. For this experiment, we 
crossed a stacked NOLH for the seven continuous factors with a full factorial for the 
categorical variables, resulting in 792 design points. Each design point was run ten times 


for a total of 7,920 runs. 
3. Data Mining and Meta Models 


Data mining is defined by Hand, Mannila, and Smyth “as the analysis of 
observational data sets to find unsuspected relationships and to summarize the data in 
novel ways that are both understandable and useful to the data owner” (Hand, 2001). The 
primary approach used for assessing PSOM once the data is “grown” is the use of meta 
models. This study followed Kleijnen and Sargent’s procedure for fitting the meta model 
(Sargent, 2000). A meta model is a an approximation of the input and output data within 
a simulation (Sargent, 2000). It is merely a simple function used to approximate a very 


complicated simulation. 


By simplifying the simulation we can gain tremendous insight into the actual 
nature of PSOM. For example, this study uses tools such as polynomial regression 
models to simplify the very complex inner workings of PSOM. Because the meta models 
used are not meant as prediction models, but rather to gain inference on the model, the 
meta models overall accuracy is not the overall goal, but gaining insight is. The primary 
types of meta models used are least squares regression and logistic regression. For much 
of the validity measures of the meta model, absolute relative error and R-Squared are the 
primary measures used in determining usefulness. We also used neural networks as a 
second modeling technique to merely validate the regression models. We assess the 
significance of the regressors in the meta model and reduce the overall number of 
regressors using the concept of Occam’s Razor; remembering that although we can 


closely fit the model and minimize error, we can lose sight on which parameters are truly 
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influential to PSOM. After the meta model has shown light on key factors, further 
analysis into the raw data and the mathematical algorithms within PSOM can be 


conducted. 
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V. DATA ANALYSIS 


Information is data that has been given meaning by way of relational 
connection. 


(Ackoff, 1989) 


By leveraging design of experiment with powerful data farming tools we are 
provided a tremendous amount of data which is not yet tractable for a simulation that has 
no true closed form solution. Although much of the “heavy lifting” is complete, we still 
need to put the tremendous amount of data into a manageable form and quantitatively 
describe its relevance. The purpose of this chapter is to describe the processes used to 
analyze the data. This analysis is formatted around gaining insight into the original 
questions posed in the beginning of this thesis. We demonstrate that by using data 
models and algorithmic models (neural networks) in conjunction with our experimental 
designs, valuable conclusions can be gained about very complex simulation models. At 
the conclusion of this process, key points about the nature of PSOM are presented. These 


conclusions are discussed in detail in Chapter VI, less the detailed mathematical analysis. 
A. DATA COLLECTION 


In line with its purpose as a human-in-the-loop wargame, the output for PSOM is 
primarily graphic in nature and meant to be displayed on a map of the area of operations. 
These briefing maps are color coded, and for the MOEs security and consent range from 
0 to 10 (see Figure 11). Fortunately, PSOM does have the ability to output data by grid 
square across the area of concern. These data are output into a comma separated value 
file (CSV) in their raw form without having been transformed to the fore mentioned 
normalized scale. Each CSV file contains the output for an individual MOE by faction or 
ethnic group in a single run. Therefore, each design of experiment provides over 100,000 
output files. Because the primary emphasis of this study is limited to the Sunni consent 


and security MOEs, the data concerning these values is an aggregate of the mean 
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response of each region based on coalition forces’ areas of responsibility and the entire 
country. The following example describes the mean consent values of the Sunni toward 


the coalition forces across all of Iraq, 


7 y Se SunniConsent 


Coalition, Iraq k a n 


Coalition 





MeanSunniConsent 


Where k = the total run number within the design point. 


n=the number of grids squares in the scenario. 


A similar method was used to look at the level of variance throughout each AOR 


and the entire country. 


k n on, 2 
mA Ds >, (x; ; _ Keoutiioninaa 


SunniConsent Variance, aition,tragq (k*ny a1 





Where x is the consent for coalition in grid square i on run j and n, the total 
number of grid squares, is multiplied by k, the number of runs of the design point. This 
value provides insight into the overall volatile nature of the responses across the area of 


concerm. 


This technique allows for further analysis into the six regions of the country 
without the overwhelming resolution of 135 responses. This permits significant enough 
granularity in the outcomes to raise a flag when outcomes are interesting, ensuring further 
analysis when needed. At this point, either analysis can take place on specific region, 
individual grid squares, or the game can be played and observed with the interesting 


parameters from the specific corresponding design point. 


Arguably this technique is not ideal. Measures of effectiveness are measured by 
geographic location and not by population. Therefore, there is no appropriate weighting 
for population size in a particular area. For example, region 4, which is the highly 
populated city of Baghdad, consists of only for two squares, which is the same number of 
squares as Samarra, which only has a 10% of the population of Baghdad. This is an 
acknowledged weakness in this study; however specific cells and regions were addressed 
when required (we played over 100 games step-by-step throughout this process). 

a2 


Initially, this entire process was conducted on a small scale design. This provided 
initial insight into the stochastic nature of PSOM, and some initial factor analysis as well 
as verifying the functionality of the computational tools developed. Although the initial 
experiment was meant to be a test, some significant insight was gained. The focus of this 
design was to demonstrate the significance of time step and faction stance. The factors in 
this experiment are displayed in Table 8. In this case, each simulation was run for 12 


time steps. So the simulation was two years, one year, or 12 weeks in simulation time. 



































IGO Stance Coalition Sunni Time Step 
Stance Nationalist 
Stance 
Level 1 Active Providing Aid | Providing 7 Days 
Level 2 Withdrawn Attacking Attacking 30 Days 
Level 3 Securing 60 Days 
Table 8. _ Initial Test DOE Factors 


Each design point was run 30 times to ensure variance could be accurately 
estimated. The results show that although variance exists between each of the 30 
iterations, it is minimal. Table 9 shows the quantiles of mean Security and variance by 
grid squares for a randomly drawn excursion file. It is important to note is that although 
the security values cover a large portion of the 1 to 10 domain the variance between the 


30 iterations by grid square is minimal. Consent results are similar. 
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Quantiles for Mean Security by Quantiles for Security Variance (#*) of 30 











Gridsquare 
N=135 runs by GridSquare N=135 
Quantiles Quantiles 
9.5050 0.23809 
9.5050 0.23809 
8.2720 0.03529 
8.0924 0.00209 
8.0787 0.00200 
7.5153 0.00166 
5.3991 0.00067 
5.3754 0.00041 
3.9384 0.00034 
2.1749 0.00031 
2.1749 0.00031 


























Table 9. — Looking at the Means and Variance of PSOM Output for Security 


The largest variance of .24 (Standard Deviation = .48) is from grid location (9,7), 
and although this value is large in comparison to the rest it is 5% of the overall range 


covered and arguably negligible when looking at the entire country of 135 squares. 


Using a least squares quadratic linear regression with two-way interactions built 
by stepwise regression we created a meta model to determine the significance of the fore 
mentioned factors in respect to the response variable Iraq mean security. Our primary 
concern in this analysis is to ensure that 12 time steps, whether they equate to 2 years or 3 
months do not provide similar results. The scaled estimate graph in Figure 12 shows the 
least square regression meta model and that time step is indeed the most significant factor 
of the three (stances are further analyzed in the following sections). The model shows 
that the 7-day time step has the most significant affect on security followed by the 30-day 
and 60-day time step. This provides more knowledge, in that as expected, the longer 
units are active in a particular area the less marginal impact they have on security. The 


model demonstrates the potential to reflect a point of diminishing returns. 
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Scaled Estimates of the Regression resulting in an R-Squared of .82 and Root Mean 


Squared Error of .41 








| Scaled Estimates 





Continuous factors centered by mean, scaled by range/2 
Scaled 
Estimate Std Error tRatio —_ Probsjt] 


Coalition Stance{2&1-0} 


Coalition Stance{2- 1} 
Sunni Stance{1-0&8} 
Sunni Stance{0-8} 




















Figure 12. Explanation of the Test DOE Regression Model 


Analyzing the data by the average security across Iraq against time step from 
PSOM we can see the significance of time step. Figure 13 shows the results of a 
Kruskal-Wallis Test comparison of security in regard to time step and a quantile plot of 
security by time step. The Kruskal-Wallis Test is a non parametric test for comparing 
multiple populations where the null hypothesis is the populations are equivalent. First, 
looking at the P-Value for the hypothesis test there is a statistically significant difference 
between security when subdivided by time step. The quantile plot visually depicts the 
test results. Figure 13 shows the substantial difference in security between a 7-day time 
step and a 30-day time step and just, as the meta model states, the difference is not as 
significant between the 30-day and 60-day time step. For a fixed number of steps, larger 


time steps results in greater security in this scenario. 


ao 





Kruskal-Wallis Test 


Normal Quantile Plot (Time Step 
Regard to Security) 


in 





(Mean- 
Count Score Sum Score Mean Mean0)/Std0 
3844.00 53.389 
8260.00 114.722 
11332.0 157.389 


Level 





7 72 
30 72 
60 72 


1-way Test, ChiSquare Approximation 
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ChiSquare DF Prob>ChiSq 
100.7895] 2 <,0001* 
Mean Security Iraq 
Figure 13. Difference in Security Determined by Time Step 


B. SCENARIO SPECIFIC DESIGN OF EXPERIMENT 


The scenario specific DOE covers those factors which are either underlying 


assumptions in the specific scenario or can be inputted by the players themselves as 


discussed in Chapter IV. 


1. Consent 


Consent is defined as the degree to which the population supports and submits to 


the will of a faction. It is a result of the difference between expectations of an ethnic 


group and the actual provision (including security) generated by a particular faction. 


Figure 14 is a correlation matrix which substantiates the use of the mean overall Sunni 


consent for coalition forces. The mean overall consent and the consent of each sub region 


show a very strong positive correlation. 
























































Variable By Varaible Correlation | Significance Prob 
SUNNI AVG CoalitionConsent Region | Mean | 0.98 0 
SUNNI AVG CoalitionConsent Region 2 Mean | 0.99 0 
SUNNI AVG CoalitionConsent Region 3 Mean | 0.99 0 
SUNNI AVG CoalitionConsent Region 4 Mean | 0.90 0 
SUNNI AVG CoalitionConsent Region 5 Mean | 0.98 0 
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Figure 14. Correlation of Mean Consent by Region and Mean Consent of Iraq for the 
Sunni Population Agents toward the Coalition 


Worth noting is the decreased correlation between the overall average consent in 
Iraq and that of Region 4. Region 4 is Baghdad, which due to its large population and 
larger amount of coalition units has a bit more extreme activity. Region 4 consent is 


isolated and analyzed later in this chapter. 


Because initial consent is a factor in the experiment, the meta model response is 
the difference between the beginning and the end of the simulation run (Consent Final- 
Consent Initial). An initial look at this response variable (Figure 15), shows that 
regardless of factor settings mean Sunni consent for the coalition did not increase for any 
of the 3,120 design points. This was a troublesome point throughout the initial 
experiment. Consent toward a faction is a function of the goods created, to include 
security, by a particular faction and the corresponding marginal gains of the population 
agents. It took multiple iterations of the design process to actually create goods because 
goods production is a balance of infrastructure produced and human capital produced. 


Once this balance is met, which it was for this DOE, production increased. 


a7 





Difference in Consent 





100.0% maximum — -0.135 


99.5% -0.536 
97.5% -0.715 
90.0% -1.245 


75.0% quartile  -2.108 
50.0% median = -3.444 
25.0% quartile  -6.069 


10.0% -7.819 
2.5% -8.455 
0.5% -8.699 


0.0% minimum —_-8.699 


Figure 15. Histogram and Quantiles of the Difference in Initial Sunni Consent and 
Final Sunni Consent 


The next step is to create a meta model to gain some insight into what factors are 
contributing to the change in Sunni consent toward the coalition. This meta model is a 
quadratic fit least squares model allowing for two-way interactions. Stepwise regression 
in JMP 7 resulted in an adequate model introducing relatively few factors. Figure 16 is 
comprised of three separate figures. The summary of fit shows an R-Square of .947, 
which is a statistic describing the proportion of the variance accounted for by the meta 
model. Because of the nature of the R-Square statistic this study carefully considers the 
number of factors in the model in conjunction with a particular R-Square with the goal 
being to explain as much variation in the model as possible with a minimum number of 


factors. Figure 16 shows that with an R-square of .94 and a mean square error of .35 our 
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model is clearly significant. The actual by predicted plot shows a fairly linear 


relationship, suggesting the meta model is sufficient for analysis of these factors and this 






















































































response. 
| Summary of Fit | | Analysis of Variance { Actual by Predicted Plot 
Sum of 07 
Source DF Squares Mean Square F Ratio 14 
Model 12]  16293.807 1357.82|_3919.297 _ 24 
Root Mean Square Error Error 3107]  1076.402 0.35] Prob >F © $34 
Mean of Response 4, C.Total | 3119] 17370.208 0.0000" 82 44 
ot 4 
Se 4 
£ g ae 
9 § 64 
74, 
-84. : 
Peeper 
-8.0-7.0 -6.0-5.0-4.0  -2.0-1.0 0.0 
Difference in Consent Predicted 
P<,0001 RSq=0.95 RMSE=0.5412 
Figure 16. Summary Information of Consent Meta Model 


Figure 17 displays the factors introduced into the meta model (less the intercept). 
The effects are sorted by the absolute value of the t-ratio, showing the most significant 


effects at the top. A bar graph shows the t-ratio, with a line showing the 0.05 significance 














level (JMP 7 Help). 
| Sorted Parameter Estimates 
Term Estimate Std Error_t Ratio Prob>it| 
Normalized Inititial Consent -0.72856 A -171.2 0.0000* 
Sunni MG Security -11.53384 ; -96.26 0.0000* 
(Sunni MG Security-0.45003)*(Normalized Inititial Consent-5) -3.274689 A -72.00 0.0000* 
(Sunni MG Security-0.45003)*(Sunni MG Security-0.45003) 81.535595 ; §2.25 0.0000* 
Coalition Stance{1&2&8-0} -0.260095 A -21.05 <.0001* 
(Sunni MG Security-0.45003)*(Sunni Marginal Gains-0.45003) 32.613802 ; 18.89 <.0001* 
Sunni Marginal Gains -1.81181 : -15.12 <.0001* 
(Coalition Force Protection-3.00615)*(Sunni Force Protection-3.00615)| | -0.110728 A -11.62 <.0001* 
Coalition Force Protection -0.088272 . -9.81 <.0001* 
Sunni Force Protection 0.0759253 A 8.44 <.0001* 
Coalition Stance{1-2&8} -0.094726 F -7.34 <.0001* 
Coalition Stance{1-2&8}+0.25)*(Sunni Marginal Gains-0.45003 0.8352763 <.0001* 




















Figure 17. Scenario DOE Sorted Parameter Meta Model 


For adjudication of our model, we also use the neural network variable 
importance tool from the Clementine software package. Figure 18 is developed using a 
Bayesian Neural Network. This technique is used throughout the thesis to ensure that we 
are focused on the significant factors. As can be seen by comparing Figures 16 and 17, in 
the two completely different modeling types the same factors (marginal gains, initial 


consent, and coalition stance) are significant. 
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Variable Importance 


Targets: Overall_AVG_Sunni_Consent 


Sunni MG Security 
Normalized Inititial Consen 
Sunni Marginal Gains 
Coalition Stance 

Sunni AIF Stance 
Coalition Reputation 
Coalition ROE 

Sunni Force Protection 
Coalition Leadership 
Coalition Cas Tolerance 
Coalition Force Protection 
Coalition Experience 
Coalition Turns at location 
Sunni AIF ATK US 


Sunni Political Ideology 
NGO Stance 
Coalition intel RelationShip with .. 





ationShip with Sunni AlFy Sunni MG S 











Figure 18. Variable Importance for the Response Consent from the Scenario DOE 
found from Using a Bayesian Neural Network Produced Using Clementine 


From Figures 17 and 18 we can gain many insights about the consent MOE. First 
off, just looking at the factors without interactions, the initial consent is the most 
significant factor followed by the Sunni population’s marginal gain for security. Sunni 
marginal gain for security also enters the model as a quadratic term indicating its 
importance to the model. Arguably this makes sense; however, these are both 
assumptions made in the scenario file which appear able to drive the game. Coalition 
stance is a significant factor; however, it is not nearly as influential as the previously 
mentioned assumptions. Sunni marginal gains for goods less security is also introduced 


as a significant factor. Finally, faction force protection values show significance. 


Looking more closely into the relationship between initial consent and the Sunni 
marginal gain for security we see the significance of these two factors in relationship to 
the final average consent of the Sunni population to the coalition. Figure 19 is a contour 
plot with the Sunni marginal gain for security on the x-axis, normalized consent on the y- 
axis, and colors from red to blue representing the corresponding values of the final 


average Sunni consent toward the coalition at each design point. 
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Contour Plot for SUNNI_AVG_CoalitionConsent 


SUNNI_AVG_CoalitionConsent 
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Sunni MG Security 
Figure 19. Contour Plot of Sunni Marginal Gain for Security and Initial Consent vs. 


Final Sunni Consent toward Coalition [Best viewed in color] 


This is very interesting because, regardless of faction settings, if the Sunni MG 
for security is .5 or greater consent for the coalition will fall below 1 in this scenario. We 
can also see the significance of the initial assumption of consent. When the Sunni 
marginal gain for security is set at a relatively low value, greater initial consent results in 
greater final consent. Also of note is the relative jaggedness of the contour plot, which in 


part is caused by the influence of other factors. 


When replaying parts of the game individually production does increase, which is 
directly tied to consent. This leads to the need for a more detailed analysis of consent in 
part 1(a) in this section. Consent has been the more complicated of the MOEs to look at. 
In PSOM it is largely a function of production level of the goods the population desires. 
It requires money, manpower, and effort put toward production. However, if consent is 


low, it is difficult to hire manpower. 


The importance of the assumptions within the scenario is notable; however, 
PSOM is a wargame, and the influence the players have over consent is important. 
Figure 20 is a bar graph comparing final Sunni consent toward the coalition in regards to 


Sunni Nationalist stance and coalition stance. The graph shows that regardless of Sunni 
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Nationalist stance, coalition consent can reach its highest points when its stance is zero 
(providing humanitarian aid). But what is more surprising is that the coalition consent 
when withdrawn is equivalent regardless of Sunni Nationalists stance. Just as surprising 
is that whether the coalition is providing security or in an attacking stance we see about 


the same consent levels. 
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Figure 20. Final Consent for the Coalition with Respect to Sunni Nationalist and 
Coalition Stances 


Figure 21 displays the consent toward coalition in regard to coalition stance. The 
green diamond provides a 95% confidence interval for the sample mean of consent at the 
particular stance. Conducting a non parametric Kruskall Wallis test to determine which 
stances matter, we see that stance 0 (provide humanitarian aid) has a significantly higher 
overall median than the other stances. This coincides with the definition of consent. 
However, the consent resulting from the other stances shows no significant difference. 
Therefore, it does not appear to matter whether all the coalition forces are securing, 


attacking, or withdrawn, which is surprising. 
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Oneway Analysis of Overall_ AVG Sunni_Consent By Coalition Stance 
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Level Minimum 10% 25% Median 75% 90% Maximum 
0 1.38e-16 0.005899 0.136487 1.707176 3.346244 8.050972 
1 8.92e-32 8.17e-10 6.182e-5 0.041186 0.639126 1.968212 7.193918 
2 1.11e-23 4.073e-6 0.00115 0.152891 1.027896 2.103971 7.298724 
8 1.43e-17 0.00031 0.011284 0.236626 1.174931 2.618229 7.614114 








Wilcoxon / Kruskal-Wallis Tests (Rank Sums) 


(Mean- 
Level Count Score Sum Score Mean Mean0)/Std0 
0 780 1493903 1915.26 12.700 
1 780 980269 1256.76 -10.874 
2 780 1135757 1456.10 -3.738 
8 780 1258831 1613.89 1.911 





1-way Test, ChiSquare Approximation 


ChiSquare DF Prob>ChiSq 
222.8762 3 <.0001* 


Figure 21. Means Diamond Plot of Sunni Consent toward Coalition in Regards to 
Coalition Stance and a Kruskal Wallis Test for Significances in the Factor Stance 


a. Increasing Consent for Coalition 


The fact that the average consent across Iraq never increased proves 
provoking and requires further attention. Looking at the separate regions of Iraq and then 
at the actual grid squares there are individual grid squares where consent increases. 
Focusing on Region 4, Baghdad, this becomes evident. There are 118 of the 3,120 total 
design points in which consent toward the coalition increased in Region 4. Figure 22 


displays the histogram of change of consent toward the coalition in the region. 
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75.0% quartile -1.419 
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Figure 22. Region 4 (Baghdad) Difference in Initial and Final Consent 


By replaying a few of these design points where consent increased it 
became evident that production also increased primarily in Region 4. Intuitively, this 


means that the coalition stance is set at 0 (humanitarian aid). 
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A nominal logistic regression meta model depicts the probability based on 
particular factors that the Sunni consent value in Region 4 increases or decreases. 
Because this model is only used to gain inference the focus is only on the factors 
currently known to be significant. The resulting meta model paints a picture of which 
factors are important to providing an increase in consent. From the parameter estimates 
below we learn that a coalition stance of 0 (humanitarian aid), combined with a high 
Sunni ROE (restrictive) and low Sunni marginal gains, will provide a high probability for 


increased consent. 

















| Nominal Logistic Fit for Region 4 Binary ] 
| Whole Model Test ) 
Model -LogLikelihood DF ChiSquare Prob>ChiSq 
Difference 277.85271 6 555.7054 <.0001* 
Full 224.32584 
Reduced 502.17855 
RSquare (U) 0.5533 


Observations (or Sum Wgts) 3120 
Converged by Gradient 


Parameter Estimates 


Term Estimate Std Error ChiSquare Prob>ChiSq 
Sunni AIF Stance[1] 1.18905319 0.1374393 74.85 <.0001* 
Coalition Stance[0] -7.6857738 1.1518985 44.52 <.0001* 
Coalition Stance[1] 5.89511525 35.514447 0.03 0.8682 
Coalition Stance[2] 5.89511525 35.514447 0.03 0.8682 
Sunni ROE -0.8161736 0.1248851 42.71 <.0001* 
Sunni MG Security 19.0823207 2.0982498 82.71 <.0001* 
Sunni Marginal Gains = 9.82317705 + 1.5671129 39.29 <.0001* 


For log odds of 0/1 


Figure 23. Nominal Logistic Regression Model Statistics for whether or not Consent 
Increased or Decreased in Baghdad 


The prediction profiler in JMP 7 is an interactive tool that allows analysis 
of the response variable (consent) due to changes in the model’s regressors. Figure 24 
depicts the prediction profiler for the nominal logistic regression model with the above 
factors. The values in the y-axis are the probabilities of consent increase. Figure 24a 
shows that if the Sunni AIF stance is 1| (attack), regardless of coalition stance, the 
probability is very low that there will be an increase in consent. Figure 24b shows the 


drastic change in the probability of consent increase when the Sunni stance is changed to 
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withdrawn (all other factors are held constant). Figure 24c increases (tightens) only 


Sunni ROE, which in turn increases the probability of an increase in consent. Figure 24d 


shows the extreme effect that changing Sunni marginal gains security has on the model. 


With a slight increase in the factor the probability of increasing consent is marginal. 


Looking at Figure 24a and comparing Sunni marginal gains and Sunni marginal gains for 


security, we can see that the functions are similar and have similar effects on the model. 
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Figure 24. Effects Profiler for Logistic Regression Model Describing Probability 


Sunni Consent toward Coalition Increases Region 4 
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Now, looking back at the raw data for the change in Region 4 consent, we 
see that it matches our model. We primary see consent increase when the coalition 
provides humanitarian aid while the Sunni Nationalists are withdrawn. Interestingly, the 
second best combination for the scenario is when the coalition is also withdrawn as 


shown in Figure 25. 
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Figure 25. Region 4 Change in Sunni Consent toward the Coalition in Regards to 


Coalition and Sunni Nationalist Stance 


Figure 26 is used to verify the accuracy of the model and to determine if 
the inference is correct about the Sunni marginal gains and Sunni marginal gains for 
security. The graphs are both histograms of the 118 data points in which consent for the 
coalition increased in Region 4. These same charts, if they were displayed using the 
original 3,120 design points, would be uniform. However, these histograms both tend 
toward the lower values of the factors. It is apparent that if a scenario is to be functional 
in the consent category, these values should be set relatively low. The exceptions to this 
conclusion are easily explainable after further analysis. In regards to Sunni marginal 
gains, there are twelve points on the histogram which have this factor at a higher level. 


The corresponding marginal gain for security for each of these design points is at its 
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lowest value of .3, which apparently is influential enough to offset the marginal gains. 
The few values at the Sunni marginal gains for security that are at the factor’s higher 
level have the lowest level of initial consent, the Sunni Nationalists, withdrawn, and a 
very high Sunni ROE value which, when all align, allow for a very small (.16 on a scale 


of 1 to 10) increase in consent. 
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Quantiles Quantiles 
100.0% maximum 0.57700 100.0% maximum 0.58100 
99.5% 0.57700 99.5% 0.58100 
97.5% 0.57700 97.5% 0.58100 
90.0% 0.55300 90.0% 0.58100 
75.0% quartile 0.42700 75.0% quartile 0.44100 
50.0% median 0.33300 50.0% = median 0.40800 
25.0% quartile 0.31400 25.0% quartile 0.36100 
10.0% 0.30000 10.0% 0.31400 
2.5% 0.30000 2.5% 0.30500 
0.5% 0.30000 0.5% 0.30500 
0.0% — minimum 0.30000 0.0% = minimum 0.30500 
( Moments } Moments 
Mean 0.3773983 Mean 0.4094915 
Std Dev 0.0833049 Std Dev 0.0733272 
Std Err Mean 0.0076688 Std Err Mean 0.0067503 
upper 95% Mean 0.392586 upper 95% Mean 0.4228602 
lower 95% Mean 0.3622106 lower 95% Mean 0.3961229 
N 118 N 118 

Figure 26. Histograms and Quantiles of Marginal Gains Values for 118 Points in 


which Consent Increased in Bagdad 


b. Consent for the Iraqi Government 


One of the key measures of a legitimate government is that it has the 
consent of the population (U.S. Army, 2006). It is important to see what effects the 
coalition can have on the consent of the Iraqi government remembering that all 
humanitarian aid by the coalition targeted the Iraqi government, giving the Iraqi 


government partial credit for the service. 
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A first look at the data (Figure 27) shows that 99% of the time consent 
dropped (starting value was 3.6); however, we were able to increase it a few times. It is 
also important to note that coalition consent often was driven to zero, but this was not the 


case for the government as shown in Figure 27. 
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Figure 27. Histogram and Quantiles Sunni Consent Across Iraq toward Iraqi 
Government for Scenario DOE 


Figure 28 shows the results of a quadratic least squares regression model 
allowing for two-way interactions with the response consent for the government. The 
model is limited to three factors: Sunni marginal gains security, Sunni marginal gains, 


and coalition stance, with a resulting R-squared value of 0.91. 
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| Summary of Fit | Analysis of Variance 





RSquare 0.908138 Sum of 

RSquare Adj 0.907932 Source DF Squares Mean Square F Ratio 
Root Mean Square Error 0.198757 Model 7 = 1215.3453 173.621 4394.998 
Mean of Response 1.207476 Error 3112 122.9370 0.040 Prob>F 
Observations (or Sum Wgts) 3120 C. Total 3119 §1338.2823 0.0000* 








Sorted Parameter Estimates 





Term Estimate Std Error 
Sunni MG Security -6.199108 
(Sunni MG Security-0.45003)*(Sunni MG Security-0.45003)} | 40.063725 
Sunni Marginal Gains -0.913203 


Coalition Stance{1&8&2-0} -0.089487 
Coalition Stance{1&8-2} -0.042026 
Coalition Stance{1-8}*(Sunni Marginal Gains-0.45003) 0.453871 
Coalition Stance{1-8 -0.015418 














Figure 28. Regression Meta Model for Sunni Consent for the Iraqi Government 


Once again, Sunni marginal gains for security prove to be the most 
significant factor. Looking at a contour plot of Sunni marginal gains for security and 
Sunni marginal gains (Figure 29), there is a strong relationship between consent for the 
Iraqi government and these factors. Sunni MG for security can dominate consent for the 


Iraqi government just as with the coalition. 
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Figure 29. Contour Plot of Sunni Marginal Gain for Security and Sunni Marginal 


Gains for Good versus Final Consent for the Iraqi Government 
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The red line in Figure 30 is a quadratic fit of government consent in 
regards to only Sunni marginal gains for security. This factor alone accounts for 87% of 


the model’s variation. 


Bivariate Fit of Consent for Iraqi Gov't By Sunni MG Security 
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[ Polynomial Fit Degree=2 








Consent for Iraqi Gov't = 3.6869191 - 6.1981582*Sunni MG Security + 
40.065422*(Sunni MG Security-0.45003)*2 

{Summary of Fit 

RSquare 0.87194 

RSquare Adj 0.871858 

Root Mean Square Error 0.234483 

Mean of Response 1.207476 

Observations (or Sum Wgts) 3120 


{Lack Of Fit 








Sum of F Ratio 

Source DF Squares MeanSquare 45.6798 
Lack Of Fit 62  82.44642 1.32978 Prob >F 
Pure Error 3055S 88.93395 0.02911 0.0000* 
TotalError 3117: 171.38036 Max RSq 
0.9335 





Figure 30. Bivariate Normal Fit: Sunni Consent toward Iraqi Government (Response) 
and Sunni Marginal Gains Security (Regressor) 


Figure 31 shows a bar graph of Iraqi government consent in regards to 
coalition stance and the corresponding non parametric test for significance of the stances. 
The graph shows that the highest values of Iraqi consent occur when the coalition stance 
is 0 (humanitarian aid). This reflects a conscious decision in the scenario setup to credit 
the Iraqi government with coalition production proving this aspect of the game is 
effective. We also see that when the coalition is providing security (level 2) we are able 
to get a slightly higher consent toward the Iraqi government than when attacking (level 1) 
or withdrawn (level 8), which makes sense as well. Based on the non parametric test 


there is a statistically significant difference among the stances in regard to consent. 
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Oneway Analysis of Consent for Iraqi Gov't By Coalition Stance 


Consent for 
Iraqi Gov't 








Coalition Stance 











{ Quantiles 


Level Minimum 10% 25% Median 75% 90% 
0 0.625591 0.672831 0.743142 1.051825 1.814224 2.619253 
1 0.617672 0.651367 0.669306 0.754121 1.422482 2.221991 
2 0.625521 0.665042 0.691267 0.95429 1.603556 2.274218 
8 0.621977 0.655839 0.686703 0.84417 1.495726 2.146167 


Wilcoxon / Kruskal-Wallis Tests (Rank Sums) | 


(Mean- 
Level Count Score Sum Score Mean Mean0)/Std0 
0 780 1391622 1784.13 8.006 
1 780 1058925) 1357.60 -7.264 
2 780 1248359) 1600.46 1.431 
8 780 1169854) 1499.81 -2.173 


| 4-way Test, ChiSquare Approximation | 


ChiSquare DF Prob>ChiSq 
92.7207 3 <.0001* 









































Figure 31. Iraqi Government Consent in Regards to Coalition Stance and the 
Corresponding Non Parametric Test for Significance of the Stances 


The correlation between the change in coalition consent and Iraqi 
government consent is shown in Figure 32. It is substantial, which is logical, as the 


coalition and the Iraqi government are allies in this scenario. 
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| Multivariate 
| Correlations 








Change in Coalition Consent Change in Gov't Consent 
Change in Coalition Consent 1.0000 0.5124 
Change in Gov't Consent 0.5124 1.0000 
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Figure 32. The Correlation between the Change in Coalition Consent and the Change 
in Iraqi Government Consent from the Sunni Population 














2. Security 


Security is defined as the perceived risk of violent death of a population agent. It 
is a function of the level of criminality, civilian casualties, and collateral damage. 
Looking at a scatter plot matrix, we see that security across the entire region of Iraq is 
correlated with each individual region, and therefore we can use the overall average as a 
good initial statistic for analysis. Although the overall average Iraqi security level is not 
as strongly correlated as consent, it is important to note that this includes the entire 
country and all ethnic groups. The lowest region correlation is .75, which is still 


significant enough to use the average security across Iraq as a response variable. 


q3 





Correlations 





Region Mean 1 Region Mean 2 Region Mean 3 Region Mean 4 Region Mean 5 Region Mean 6 AVG_Iraq Security 


Region Mean 1 1.0000 0.9526 0.7393 0.7489 0.8386 0.7777 0.9362 
Region Mean 2 0.9526 1.0000 0.8845 0.8548 0.8157 0.7228 0.9387 
Region Mean 3 0.7393 0.8845 1.0000 0.8314 0.5938 0.4587 0.7581 
Region Mean 4 0.7489 0.8548 0.8314 1.0000 0.6876 0.5957 0.7995 
Region Mean 5 0.8386 0.8157 0.5938 0.6876 1.0000 0.9850 0.9634 
Region Mean 6 0.7777 0.7228 0.4587 0.5957 0.9850 1.0000 0.9136 
AVG Iraq Security 0.9362 0.9387 0.7581 0.7995 0.9634 0.9136 1.0000 
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Figure 33. Correlations of Security by Region and the Overall Country of Iraq 


Histograms of the security output show that security after twelve months has a 
substantial dispersion. Security started at 5.8 for this scenario, so it has both increased 


and decreased in this DOE. 
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{ Quantiles } 
100.0% maximum 7.2022 











99.5% 7.0124 
97.5% 6.7751 
90.0% 6.3209 
75.0% quartile 5.8981 
50.0% median 5.3414 
25.0% quartile 4.6550 
10.0% 4.1119 
2.5% 3.6471 
0.5% 3.4669 
0.0% minimum 3.0728 

| Moments 

Mean 5.279968 
Std Dev 0.8312745 
Std Err Mean 0.0148822 


upper 95% Mean _ 5.3091479 
low er 95% Mean _ 5.2507881 
N 3120 


Figure 34. Histogram and Quantiles of Final Security in Iraq for Scenario DOE 


74 


Next, using a quadratic stepwise linear regression allowing for interactions, we 
get a telling model. Once again, we look for the point at which adding factors into the 
model yields minimal improvement. This results in an R-squared of 0.76, with only nine 
of the original twenty one factors entering the model. Figure 35a shows the summary of 
fit and Figure 35b is the scaled estimates description of the model from JMP 7. This 
output shows not only the model, but nicely displays the scaled effect each regressor has 
on the response (security). Bars that are to the left have a negative effect on the response 


and bars to the right have a positive effect. 























































































































a. { Analysis of Variance ) Actual by Predicted Plot 
Sum of 

| Summary of Fit Source DF Squares Mean Square F Ratio 7 

Model 18  1643.8997 91.3278 553.8065 3 
RSquare 0.76273 Error 3101 511.3833 0.1649 Prob >F g 26 
RSquare Adj 0.761353 C. Total 3119  2155.2830 0.0000* =| < 
Root Mean Square Error 0.40609 g 5 5 
Mean of Response 5.279968 8 
Observations (or Sum Wgts) 3120 

AVG _Iraq Security Predicted 
P<.0001 RSq=0.76 RMSE=0.4061 
b. 

{ Sorted Parameter Estimates } 
Term Estimate Std Error t Ratio Prob>|t| 
Coalition ROE 0.2419903 0.006204 39.01 <.0001* 
Sunni Marginal Gains -3.190099 0.082689 -38.58 <.0001* 
Sunni AIF Stance[1]*(Coalition Stance{2-8&0}+0.25) 0.3659304 0.00958 38.20 <.0001* 
Sunni ROE 0.2049923 0.006204 33.04 <.0001* 
Sunni AIF Stance[1]*(Coalition Stance{2&8&0-1}-0.5) -0.265439 0.009534 -27.84 <.0001* 
Coalition Stance{2-8&0} -0.250868 0.009459 -26.52 <.0001* 
Sunni AIF Stance[1] -0.205509 0.007889 -26.05 <.0001* 
Coalition Stance{2&8&0- 1} -0.219639 0.009098 -24.14 <.0001* 
(Coalition Stance{2-8&0}+0.25)*( Coalition ROE-3.00615) 0.1140548 0.007482 15.24 <.0001* 
(Coalition Stance{2&88&0- 1}-0.5)*(Sunni ROE-3.00615) 0.1025021 0.007163 14.31 <.0001* 
(Coalition Stance{2&88&0-1}-0.5)*(Coalition Cas Tolerance-50.0308) -0.003675 0.000287 -12.83 <.0001* 
(Coalition Stance{2&88&0-1}-0.5)*Coalition intel RelationShip w ith Sunni AIF[O] 0.1055139 0.008876 11.89 <.0001* 
Sunni Force Protection 0.0711882 0.006204 11.47 <.0001* 
Coalition Force Protection -0.066782 0.006205 -10.76 <.0001* 
Coalition Cas Tolerance 0.0019135 0.000248 7.71 <.0001* 
Coalition intel RelationShip w ith Sunni AIF[O] -0.041781 0.007514 -5.56 <.0001* 
(Coalition Force Protection-3.00615)*(Sunni Force Protection-3.00615) 0.0364385 0.006554 5.56 <.0001* 
(Coalition ROE-3.00615)*(Sunni Marginal Gains-0.45003) -0.20074 0.095331 -2.11 0.0353* 























Figure 35. Meta Model Results for Scenario DOE in Regards to Security 


Immediately, the significance of ROE for both the Sunni Nationalists and the 
coalition forces becomes evident. Figure 36 shows that although there are possible 
exceptions, we see an increase in overall security as both coalition and Sunni Nationalist 


ROE increase. 
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Figure 36. Contour Plot of Sunni Nationalist’s ROE and Coalition ROE as a Function 
of Average Security in Iraq 


The next factor of significant importance is that of marginal gains. What is non- 
intuitive is that the marginal gain for security is not found in this. Looking into the 
security algorithm we found that this is primarily because in this scenario we have fifteen 
marginal gains including security, so security only has one-fifteenth the influence in the 
population’s decision making. Also, the marginal gains were limited between .3 and .6 to 


prevent extreme values of the parameter dominating the model. 


Figure 37 displays this relationship through a bivariate fit of the response security 
in regards to marginal gains. There is a clear underlying linear relationship between 
marginal gains and security. The correlation between the two is —.34. This explains the 
significance in marginal gains and the importance of the assumptions which create these 
attributes in regard to the population. However, there is still considerable variability in 


this trend. 
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Bivariate Fit of AVG_lraq Security By Sunni Marginal Gains 
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Figure 37. Bivariate Fit of Average Iraq Security (Response) in Regards to Sunni 
Marginal Gains (Regressor) 


The importance of stances for both the coalition and the Sunni Nationalists are 
also evident in the meta model. This finding corresponds with the third value in Figure 
35 which explains our meta model. Equally interesting is that the interaction of stances is 


more significant than the stances themselves. 


Based on the meta model, coalition stance is much more significant toward the 
nation’s security than toward consent. The plot of average security levels by coalition 
stances (Figure 38) provides insight into the model. The coalition stance attack (Level 1) 
provides the best security on average followed by humanitarian aid (0) and withdrawn 
(8), which are not significantly different from one another. Finally, the worst stance is 
coalition securing with patrols, which is statistically less than the others. The differences 
among the means are all statistically significant except for the differences between 
withdrawn and humanitarian aid. However, the highest values of security are found 
when the coalition is withdrawn or only providing humanitarian aid. This is a result of 


interactions between faction stances. 
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Oneway Analysis of AVG_Iraq Security By Coalition Stance 
































7 
| Level to Stance 

24 0 Humanitarian Aid 
BI 1 Attack 
z sf fT 1 2 Secure/Patrol 
S | 8 Withdrawn 

a 

a3 Mean Security is highest when the 





coalition stance is attack. 








Coalition Stance 

















( Quantiles 
Level Minimum 10% 25% Median 75% 90% Maximum 
0 3.426129 3.994911 4.632386 5.276936 5.874276 6.269468 7.13993 
1 3.43832 4.624312 5.302185 5.780587 6.206263 6.639774 7.074497 
2 3.072836 3.977545 4.377682 4.995172 5.514137 5.958448 6.847882 
8 3.427015 3.998793 4.589194 5.238212 5.856593 6.215663 7.202178 
( Wilcoxon / Kruskal-Wallis Tests (Rank Sums) ] 

(Mean- 
Level Count Score Sum Score Mean Mean0)/Std0 
0 780 1176108 1507.83 -1.886 
1 780 1584185 2031.01 16.844 
2 780 948450 1215.96 -12.334 
8 780 1160017 1487.20 -2.624 








1-way Test, ChiSquare Approximation 


ChiSquare DF Prob>ChiSq 
334.7291 3 <.0001* 





Figure 38. Means Diamond Plot of Iraq Security in Regards to Coalition Stance 


Looking at just the Sunni AIF stance in Figure 39, we see that although the 
difference is not tremendous, there is a difference in security between when the Sunni 
Nationalists are withdrawn or present. The lack of a tremendous measured difference can 


be attributed to the actual game play and is discussed in detail in the conclusion. 
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Oneway Analysis of AVG_Iraq Security By Sunni AIF Stance 
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| Quantiles 
Level Minimum 10% 25% Median 75% 90% Maximum 
1 3.25992 4.004505 4.538555 5.147196 5.714113 6.164956 7.074497 
8 3.072836 4.332493 5.161951 5.742247 6.125022 6.559891 7.202178 
Wilcoxon / Kruskal-Wallis Tests (Rank Sums) _| 
(Mean- 

Level Count ScoreSum Score Mean Mean0)/Std0 
1 2080 2883766 1386.43 -15.265 
8 1040 1984994 1908.65 15.265 

2-Sample Test, Normal Approximation | 








s Z Prob>|Z| 
1984994 15.26484 — 0.0000* 








1-way Test, ChiSquare Approximation | 


ChiSquare DF Prob>ChiSq 
233.0159 1 <.0001* 





Figure 39. Means Diamond Plot of Average Iraq Security in Regards to Sunni 
Nationalist Stance 


Looking back at the meta model, the interaction of stances is clearly significant. 
Figure 40 is an interaction plot for the factors in regards to the final level of security. The 
interaction plot provides two pieces of information. The steeper the regressors 
corresponding line, the more significant the factor, and if two regressors are not parallel, 
there is likely an interaction. Thus, when the lines are crossed this alludes to strong 
interactions between the regressors. This particular interaction plot shows strong 


interactions amongst the coalition stance and Sunni Nationalist stance factors. 
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Figure 40. Interaction Plot of Coalition and Sunni Nationalist Stances in Regards to 
Iraq Security 


The intent of Figure 41 is to tease out the significance of the interactions 
discovered in our model by a visual representation of the actual data. The graph is 
coalition stance within Sunni Nationalist stance in regards to security. We see that when 
the Sunni Nationalists are actively attacking (Stance 1) the coalition stances that increase 
security are 1 and 2 (attacking and securing respectively), which are both offensive 
stances. When the Sunni Nationalists withdraw (Stance 8) the coalition are best to either 
withdraw or provide humanitarian aid. We already know that coalition stance is 
significant in terms of the mean; however, looking at the highest security values per 


stance in Figure 40, we see this as well. 
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Figure 41. Interactions of Coalition and Sunni Nationalist Stances in Regards to 
Security [Best viewed in color] 


Also of interest is the importance of the interaction for coalition stance and ROE. 
Figure 42 is a fitted plot of the security data with coalition stance held constant at 
securing (2) or attacking (1). There is a significant relationship which is also intuitive 
between ROE and security. Tighter ROE results in higher security for both offensive 


stances. 
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Figure 42. Fitted Plot of All the Security Data with Coalition Stance Held Constant at 
Securing (2) or Attacking (1) 


The significance of force protection and casualty tolerance is also of note. 
Although not the most significant of variables, their introduction shows the depth of the 
security algorithm of PSOM. Force protection determines the risk a unit is willing to 
take in order to accomplish its mission. The meta model shows the greater the force 
protection level (which results in more aggressive contact) the lower the security value. 
However, casualty tolerance has a positive effect on security in our model. Casualty 
tolerance directly affects the enemy’s ability to deter the coalition. Therefore, a high 
casualty tolerance will ensure the coalition continues its mission cycle in that unit’s area 


of operations. 


Finally, it is important to ensure that security and consent are truly different 
MOEs, although they should have some sort of relationship. Redundant MOEs can result 


in unwarranted reward for the wargame participants or irrelevant post analysis. 
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Figure 43 shows a correlation between the mean Iraqi consent and mean Iraqi 
security. There is a slightly positive correlation between the two responses, but 
inarguably they are different MOEs. 


Correlations 


AVG Iraq Security Sunni Consent tow ard Coalition 


AVG Iraq Security 1.0000 
Sunni Consent tow ard Coalition 0.1137 





Figure 43. Correlation between the Mean Iraqi Consent and Mean Iraqi Security 


Cc. SETTINGS SPECIFIC DESIGN OF EXPERIMENT 


The settings file encompasses all the underlying assumptions that are not scenario 
specific, such as the size of an infantry company, or the attributes of a particular stance. 
This design of experiments consists of sixty-six continuous factors. Each factor is 
adjusted to + 20% of the initial value used by DSTL when creating this particular settings 
file. This prevents any one factor from dominating the experiment. When looking at the 
settings file, focusing on only the contributing variables allows this research to explore 
particular mathematical models within PSOM. The primary goals in looking at the game 
settings are to determine if PSOM can be used to assess changes to unit’s abilities and to 


determine if there are settings which, if not truly understood, can drive the game. 
1. Consent 


The first MOE used as a response is consent toward the coalition. Figure 44 is a 
scatter plot matrix showing once again that the overall average Sunni consent for 
coalition is correlated with the consent in each region and therefore is a good metric to 


use for initial analysis. 
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Figure 44. Scatter Plot Matrix Showing the Overall Average Sunni Consent for 
Coalition is Correlated with the Consent in Each Region 


Looking at a histogram of the output, and remembering the starting condition for 
the scenario is the Sunni’s having consent of 2.5 for the coalition, we see that regardless 
of settings consent does not increase. Also, the range for consent change is not very 
substantial in regards to the scale of possible outcomes. It appears that the overall 


outcome of the simulation is not particularly sensitive to the experimental factors. 
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| Quantiles | 
100.0% maximum 2.3694 
99.5% 2.3408 
97.5% 2.3120 
90.0% 2.2702 


75.0% quartile 2.2284 
50.0% median 2.1639 
25.0% quartile 2.0883 








10.0% 2.0266 
2.5% 1.9563 
0.5% 1.8905 
0.0% minimum = 1.8174 
Moments 
Mean 2.1542624 
Std Dev 0.0949538 
Std Err Mean 0.0030027 


upper 95% Mean 2.1601548 
lower 95% Mean 2.1483701 
N 1000 


Figure 45. Histogram and Quantiles of Sunni Average Consent toward Coalition 
from the Settings DOE 


Figure 46 shows the results of a stepwise least square regression meta model with 
two-way interactions. This model provides an initial look into the response of mean Iraq 
consent for the coalition based on the controlled settings factors. The most significant 
factor in the model is the mean ROE setting. This coincides with the importance of ROE 
throughout the model. Although not well defined, the author’s understanding is the mean 
ROE can be related as the expected ROE of a particular area (otherwise, why make it 
variable). The recommended mean ROE is 3 (Parkman, 2008); however, varying this 
factor can have a substantial effect on consent. Secondly, the population memory 
coefficient is a significant factor in regards to consent. The greater this value, the less 
consent changes per time step. This makes sense; however, this value is a powerful 


assumption about the population. 
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Term Estimate i Prob>|t| 
MeanROE -0.210931 : 0.0000* 
PopulationMemory Coef 0.1076177 : <.0001* 
ROEMod -0.634123 : <.0001* 
ForceProtectionMean -0.053347 : <.0001* 
UnitProtection 0.0014413 . <.0001* 
UnitFirePow er -0.001788 : <.0001* 
InterUnitBaseCasualties DEF -1.296548 : <.0001* 
UNitCollateralDamage -0.013793 : <.0001* 
(PopulationMemory Coef-3.0006)*(MeanROE-3.0006) 0.0531651 . <.0001* 
(UnitFirePow er-80.016)*(UnitProtection- 100.02) 1.5352e-5 . 0.0309* 
(ForceProtectionMean-3.0006)*(ROEMod-0.30006) -0.024138 ' 0.7067 
*(InterUnitBaseCasualties DEF-0.10002 -0.011726 : 0.8851 











Figure 46. Meta Model for Sunni Consent toward the Coalition 


Looking at a contour plot of the two most significant factors identified from the 
meta model (Figure 47), we see that although other factors are having some effect, these 
primary two factors have a tremendous impact on consent. The lowest consent values are 
obtained with a low memory coefficient and a high mean ROE. The converse is true as 


well. 
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Figure 47. Contour Plot Showing Effects of Mean ROE and Population Memory on 
Consent [Best viewed in color] 


A single factor least squares regression provides a more thorough understanding 
of the significance of mean ROE. Figure 48 shows we can obtain an R-squared of .59 


with mean ROE as the only regressor. This shows the exceptional importance of this 


factor. 
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| Bivariate Fit of OverallAVGConsent By MeanROE | 
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{ Linear Fit 

OverallAV GConsent = 2.7863077 - 0.2106396*MeanROE 
| Summary of Fit 
RSquare 0.591114 
RSquare Adj 0.590704 
Root Mean Square Error 0.060748 
Mean of Response 2.154262 
Observations (or Sum Wats) 1000 





| Analysis of Variance 





Sum of 


Source DF Squares Mean Square F Ratio 
Model 1 §.3242804 5.32428 1442.778 
Error 998  3.6829183 0.00369 Prob >F 
C. Total 999 9.0071988 <.0001* 





| Parameter Estimates | 
Term Estimate StdError t Ratio Probs|t| 
Intercept 2.7863077 0.01675 166.34 0.0000* 
MeanROE -0.21064 0.005545 -37.98 <.0001* 





Figure 48. Linear Regression Meta Model for Consent using only Mean ROE as a 
Regressor 


The importance of mean ROE suggests that it is worth looking into its use in the 
algorithms which determine civilian casualties. The following equations and their 
explanation are taken from Paragraph 167 of the Peace Support Operations Model 
Specifications Manual (Jon Parkman, 2008). These equations explain an important part 
of the combative algorithms in the simulation. The highlighted equations take into 
account the mean ROE and the ROE modifier which is used to control the level which 
rules of engagement actually impact civilian casualties (Jon Parkman, 2008). 


Mathematically sz, (mean ROE) is used in an exponential role in determining the number 


of civilians killed due to combat. 


FP = (ef -A)feiole A) 


c= By {SAS \(r,2, Jot 


Be 1s 











FP (1-1) 











Where: 


FP = Contact Force Protection value of Units 1 and 2 (0 < F< 1) 

fm = Force protection modifier (0 < fn < 1) 

f,, f. = Force protection values of Units 1 and 2 (1 <f<5) 

pre= Mean Force Protection value — (Typically 3) 

Ci, C2 = Casualties taken by Units 1 and 2 (0<C< ~~) 

Bate = Average number of casualties taken in offensive operations of size B, 
(from HA, SIGACTS and other sources) (Constant) 

Boer = Average number of casualties taken in defensive operations of size B. 
(from HA, SIGACTS and other sources) (Constant) 

B. = Average size of contact (from HA, SIGACTS and other sources) (Constant) 

Si, S2 = Size of force in contact from Units 1 and 2 (0 < S < ©) — where Unit 2 
may represent multiple defending units. 

F,, F, = Average Firepower values of Units 1 and 2 (0 <F<~-) 

P,, Pz = Average Protection values of Units 1 and 2 (0 < P< ~) 

T;, Tp = Terrain modifiers on Firepower and Protection (0 < T < 2) 

I = Indirectness level of Unit 1 (0 <I < 1) 

Rn = RoE modifier (0 < Ry < 1) 

Ry, R2 = RoE levels of Units 1 and 2 (1 <R<5) 


Cciy = Casualty level taken by civilians (0 < Cc, < ©) 


In a real world situation, it makes sense that the casualty level of civilians should 
play a key part in the consent of a population. Furthermore, the ability to adjust the 
impact is important in designing the wargame. This also shows the importance of the 
value of security. Although it is not directly involved in consent, because the majority of 
coalition forces have assumed a stance that involves providing security in this scenario, 
the ways in which the coalition affects the marginal gain of security is key. In changing 


the settings file we have either increased or decreased the number of civilians killed and 
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therefore changed the civilian perception of security which has an impact on consent. 
This is a good example of the robustness of PSOM in regards to irregular warfare 
modeling. However, the mean ROE in this game is very sensitive and can outweigh a 


strategic plan in the game if allowed. 


The memory coefficient is a scalar which determines the speed at which a 
population agent’s consent can change. The following equation, taken from the PSOM 
manual, shows part of the consent algorithm and the memory coefficient’s role: 

t-1(7,M 
(ac, +c (k” —1)) 
k™ 





Cc! 





Where: 


C; = The final consent rating for faction f in turn t 


Cc = The consent rating for faction f in turn t-1. 


k™' = The memory coefficient. (Jon Parkman, 2008) 


What becomes evident is that the larger the memory coefficient is, the less 
sensitive the change in consent will be. This includes even small changes such as the 
range covered in this experiment (2.4 to 3.6). The importance of this parameter makes 
sense because of its direct relationship with the final consent. It is crucial to understand 
that this value, if misused, can cause serious issues in the simulation. This implication is 
particularly true if model is used as a wargame over a short period of time. If players do 


not see the effects of their decisions, interest can be quickly lost. 


Looking back at the meta model, the unit attributes of firepower and protection 
are significant in regards to consent. As unit firepower increases the consent goes down. 
This is most likely due to the number of civilians killed, which links directly to the 
marginal gain value of security. As seen in the previous equations, this is a very 
simplistic look at combat interactions. Just because a unit possesses a tremendous 
amount of firepower does not mean this firepower is projected, nor does it mean this 


firepower will result in civilian casualties. 
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As the protection attribute increases consent increases because this allows for 
more deliberate focus on the mission. The first reason is the less causalities a unit takes, 
the less likely they are to be deterred from the mission. More indirectly, but the above 
equations also show, that the greater the coalition protection the more coalition survive 
resulting in fewer civilian causalities because the coalition ROE is tight. The relationship 
between consent and the unit’s collateral damage level is also negatively correlated, 


which is intuitive. 


The limited effects of the unit attribute changes provide some key insights. 
Further analysis must be conducted to prove the hypothesis that PSOM can be used to 
gain information on force capability modification. Arguably, the addition or removal of 
20% of a maneuver company’s capability should have a significant effect. 
Unfortunately, only three of the ten attributes in the experiment proved significant when 
changed +20%. This is a bit disconcerting if the model is intended to test unit 


capabilities. 
2. Security 


Looking at the data we see that security does increase. It starts at 5.8 throughout 
the country in this scenario. Figure 49 is a histogram and analysis of the security output 
showing that security increased in over 95% of the design points. Also, even though 
security is on the same scale as consent (1 to 10), the security response variable covers a 


much greater range (from 5.7 to 7.7). 
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| Quantiles | 
100.0% maximum = 7.7217 
99.5% 7.6993 
97.5% 7.5422 
90.0% 7.3703 


75.0% quartile 7.2097 
50.0% median 6.9605 
25.0% quartile 6.6839 








10.0% 6.4763 
2.5% 6.2904 
0.5% 6.1428 
0.0% minimum 5.7681 
Moments 
Mean 6.9389892 
Std Dev 0.3412168 
Std Err Mean 0.0107902 


upper 95% Mean 6.9601633 
lower 95% Mean 6.9178151 
N 1000 


Figure 49. Histogram and Analysis of the Security Output 
Figure 50 is the scaled estimate results of the least squares regression meta model 


with two-way interactions. Once again, mean ROE dominates followed by the ROE 


mod. Unit firepower and unit protection also prove significant. 








| Scaled Estimates ) 
Continuous factors centered by mean, scaled by range/2 
Scaled 

Term Estimate t Ratio Probs|t| 
Intercept 6.9389927 2595.85 0.0000* 
UnitFirePow er -0.092552 -19.97 <.0001* 
UnitProtection 0.1024573 22.15 <.0001* 
ForceProtectionMean -0.115134 -24.87 <.0001* 
MeanROE -0.511041 -110.37 0.0000* 
ROEMod -0.156403 -33.78 <.0001* 
InterUnitBas eCasualties DEF -0.095816 -20.69 <.0001* 
(UnitFirePow er-80.016)*(InterUnitBaseCasualtiesDEF-0.10002) -0.002674 -0.33 0.7407 
(UnitProtection-100.02)*(ForceProtectionMean-3.0006) 0.0171346 2.13 0.0333* 
(MeanROE-3.0006)*(ROEMod-0.30006) -0.019127 -2.41 0.0160* 





Figure 50. Non-ordered Scaled Estimates of Parameters of Settings File DOE in 
Regards to Security 
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Figure 51 is a contour plot of security in regards to the mean ROE and ROE 


modifier. Mean ROE clearly has a tremendous effect on security. 
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Figure 51. Contour Plot of Mean ROE and ROE Mod versus Security 


Referring to the security meta model in Figure 49, and focusing on unit firepower 
and unit protection, we see the same results as from earlier with consent. An increase in 
a unit’s firepower results in a lower-end state security, and an increase in protection 
results in an increase in security. This negative relationship between unit firepower and 
the responses creates a pattern in the model: the more firepower a unit has the more likely 


they are to use it. 


Because the meta models for security and consent had such similar results, further 
analysis into the relationship between security and consent for this DOE proves 
beneficial. Figure 52 is a correlation scatter plot matrix for security and consent for the 
1,000 design points in the settings file. With a correlation of .9, clearly there is a nearly 
linear relationship between security and consent for this experimental design. This can 
be attributed to the primary factor driving change throughout the experiment—mean 
ROE. Because most factors that have effects on functions other than the combative 
algorithms were not significant with the 20% interval used in this experiment, the 
combative functionality of PSOM can account for the majority of the variation in both 


security and consent. 


93 





| Multivariate 








| Correlations 
lragAVG_Security IragAVG_Consent (Coalition) 

lIragAVG_Security 1.0000 0.9054 

lIragAVG_Consent (Coalition) 0.9054 1.0000 





560 rows not used due to missing or excluded values or frequency or w eight 
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Figure 52. Scatter Plot Correlation Matrix of the Outputs Consent and Security in the 
Settings DOE 


The settings files contribute to the underlying assumptions and factors of PSOM. 
Clearly, over sensitivity of these functions could prove problematic in the use of the 
model. Of the sixty-six setting factors changed, only a handful proved sensitive to a 20% 
change from the initial values. This provides some insurance that for the most part 
questionable settings will not ruin the validity of the game. This analysis has identified 
those factors which should be carefully manipulated. However, the lack of sensitivity to 
unit attributes could prevent PSOM from becoming a tool used for comparative analysis 


of unit equipment or force structure. 
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D. CUMULATIVE DOE 


Using information gained over six months of studying PSOM and all the lessons 
learned from earlier analysis, we devised a cumulative experiment. In this experiment we 
focused on coalition stance, unit attributes, ethnic group characteristics, and time. It is 
important to remember this design used a scenario in which the Sunni Nationalists are 


actively attacking the coalition and Iraqi infrastructure. 


1. Consent toward Coalition 


In this case, consent outcomes toward the coalition are moderate compared to the 


previous experiments. Figure 53 also shows that consent increased in approximately 


20% of data points (initial value is 2.5). 
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Figure 53. Histogram and Quantiles for the Cumulative DOE in Regards to Consent 
toward Coalition 
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Figure 54 displays a stepwise quadratic linear regression with two-way 


interactions with a very small mean square error and an R-squared of 0.95. 


















































| Summary of Fit _Sorted Parameter Estimates 
RSquare 0.951374 Term - Estimate Std Error_t Ratio Prob>|t| 
‘Sunni MG Security -0.814495) 0.016333] -49.87 <.0001* 
RSquare Adj 0.94864 Coalition Stance[0}*Time in Months[12] 0.0179539} 0.000486] 36.91 <,0001* 
Root Mean Square Error 0.007902 Time in Months[12] 0.0117616] 0.000486) 24.18 <.0001* 
Mean of Response 2.474794 Time in Months[12]*(Sunni MG Security-0.1503) 0.2660761| 0.016323] 16.30 _|| <.0001* 
Observations (or Sum Wats) 264 (Sunni MG Security-0.1503)*(Sunni Marginal Gains-0.1503) -5.038668} 0.602148} -8.37 il <.0001* 
Sunni Marginal Gains 0.1056499] 0.016348} 6.46 <.0001* 
(UnitFirePow er-100.061)*(UnitSensors-100.061) 2.7978e-6| 7.314e-7 3.82 0.0002* 
(UnitFirePow er-100.061)*(UnitChangeA ttitudePerMan-5.51515)|| 1.9332e-5] 6.436e-6] 3.00 0.0029* 
(UnitSize-100.061)*(UnitChanged ttitudePerMan-5.51515) 2.0286e-5| 7.187e-6] 2.82 0.0052* 
Coalition Stance[0] -0.001035] 0.000486]  -2.13 0.0344* 
UnitFirePow er -2.176e-5] 1.634e-5|  -1.33 0.1841 
UnitChangeA ttitudePerMan 8.8625e-5| 0.000182] 0.49 0.6271 
UnitSize 7.068e-6] 1.633e-5) 0.43 0.6655 
UnitSensors -3.864e-7] 1.633e-5 -0.02 0.9811 












































Figure 54. Least Squares Meta Model Results for the Cumulative DOE in Regards to 
the Response Consent 


In this experiment we intentionally limited the range of Sunni marginal gain 


security to between .1 and .2._ However, it remains a determining factor in this model. 


The next interesting finding is the importance of the interaction of coalition stance 
and time in months. According to the model, consent toward the coalition is greater if the 
coalition is providing humanitarian aid over a 12-month period than if the collation 
provides aid over a 24-month period. This seems counterintuitive. The increase in 
production directly increases consent and over twenty-four months there is more 
production than over twelve months. However, the lack of security provided by this 
tactic over the second year seems to decrease consent toward the coalition. This shows 
the dynamics of PSOM. From this telling analysis it appears that a faction cannot just 


“build” consent. 


Analyzing the importance of time in regards to the data from the experiment, we 
see that, on average, a 12-month simulation results in a greater consent. Figure 55 (left 
side) shows a means diamond plot of consent toward the coalition at the end of both 
twelve and twenty-four months. We can see consent is greater at twelve months than at 


twenty-four. The right side of Figure 55 is a quantile plot where the red line shows each 
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data point for the 12-month design points, and the blue line shows the data point for each 
24-month design point. For all but the 99% quantile twelve months has a higher consent 


toward coalition. 
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Figure 55. Consent versus Time in Months for the Cumulative DOE 


Figure 56 shows the interaction between coalition stance and the simulation time 
in regards to consent. What is noticeable is that when the coalition is providing 
humanitarian aid (Stance 0) we see a decrease in consent from a one year simulation to a 
two year simulation. However, there is a slight increase in consent between year one and 
year two when the coalition is in a securing stance (Stance 2). This supports the meta 
model’s (Figure 54) findings that providing just humanitarian aid is not a simple path to 


increased consent in PSOM. 
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Figure 56. The Interaction between Time and Coalition Stance in Regards to Consent 


Also, we now see the lack of significance of the unit descriptor variables (Figure 
54). According to the regressor meta model the unit attributes are not significant alone. 
However, they are significant amongst themselves with interactions. These interactions 
seem intuitive. For example, the increase in unit size alone is not significant; however, 
when combined with increasing the unit’s ability to change perception, we see 
significance in these parameters’ interaction. It is important to remember these variables 
were changed with an exceptional range (+50% the original value), which may not be 


representative of reality. 


2. Security 


Looking at the average security values for each design point in Figure 57 we see 


that security always increased throughout the country (the starting value was 5.8). 


98 





| AVG Security 











——_{_oL}— 





_—————t 


















































ii 












































68 6.9 7 7.1 7.2 7.3 7.4 7.5 76 7.7 








Quantiles 








100.0% maximum 7.7325 
99.5% 7.7291 
97.5% 7.7085 
90.0% 7.6302 
75.0% quartile 7.5224 
50.0% median 7.4104 
25.0% quartile 7.1641 
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2.5% 6.8824 
0.5% 6.8489 
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Moments 
Mean 7.3455151 
Std Dev 0.2373461 
Std Err Mean 0.0146076 
upper 95% Mean = 7.374278 


lower 95% Mean 7.3167523 
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Figure 57. Cumulative DOE Histogram of Final Security Value 


Figure 58 shows the results of the quadratic least squares meta model with 


security as the response variable. The factors we adjusted accounted for a very large 


portion of the variance within the simulation. 


Summary of Fit 





RSquare 0.971378 
RSquare Adj 0.97048 
Root Mean Square Error 0.040779 
Mean of Response 7.345515 
Observations (or Sum Wgts) 264 
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Term 


Estimate 
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Figure 58. Regression Meta Model Results with Security as the Response Variable 
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From the parameter estimates it appears that, just as with consent, simulation 
time, coalition stance, and the Sunni marginal gains are significant factors within this 


experiment. 


Figure 59 shows the means diamond plot of security in regards to time along with 
the normal quantile plot of both the 12-month design points and the 24-month design 
points. From this picture, security, unlike consent, seems to increase when the scenario is 


executed over two years. 


Oneway Analysis of AVG_Security By Time in Months 





Time in Months 


Normal Quantile 





Figure 59. Means Diamond and Normal Quantile Plots of Security in Regards to 
Time 


Figure 60 shows the same data, but introduces the interaction of unit stances. In 
both cases where the collation stance is either provide humanitarian aid (0) or secure 
through patrolling (2) security increases between the one-year design points and the two- 
year design points. | However, we see a greater quantity in this increase when the 
coalition is securing. This agrees with the analysis of consent. It seems that although in 
the first year gains can be made through the use of humanitarian aid a faction must also 


focus on security. 
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Figure 60. The Interaction between Time and Coalition Stance in Regards to Security 


Looking just at marginal gains in regards to security (Figure 61) we see that there 
is a clear trend that when marginal gains are higher security is decreased. The space 
where there is no data is a result of the variables coalition stance and time. The darker 


points are the sixty-six points where the collation forces are patrolling for the one-year 


time frame. 
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Figure 61. Bivariate Fit of Security by Sunni Marginal Gains 
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The final bit of analysis conducted was to determine the stability of the model 
with respect to time step. For this experiment we simulated the initial Iraq scenario for 
one year of combat changing only the time step for each turn of the game. We found 


that, for consent, time step can have a substantial effect on the game as shown below in 


Time Step 














Table 10. 
Time Step Total number | Sunni Consent | Security 
of steps for Coalition 
7 Days=1 time step 52 1.28 PAS 
30 Days=1 time step | 12 2.19 7.23 
60 days = 1 time step | 6 2.36 PAG 

















Table 10. 


Security and Consent Responses Resulting from Changing only Time Step 


E. SUMMARY 


By leveraging intelligent design of experiments with powerful computing we have 
been able to obtain a very robust scope of output from PSOM. Using some basic data 


mining techniques we can gain some tremendous insight into the simulation model. A 


synopsis of the results is as follows: 


The methodology used within this thesis proved a useful process to 
analyze social models and produce viable results. 


The initial assumptions made about the population can be a controlling 
factor of the game’s outcome. This is especially true with respect to the 
ethnic group’s marginal gains and initial levels of consent toward a 


particular faction. 


Although it is difficult to increase consent in the game, it is not 
A careful balance of productivity and security provided 


impossible. 


should result in increased consent. 
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Although there is a stochastic aspect to PSOM, it is minimal. It is a 
wargame and variation primarily results from the human-in-the-loop 
aspect of the game. 


Time is an important contributor to the game. This applies to both the 
overall simulated time and time step between turns. There is a difference 
between twelve weeks of combat and twelve months, and this is 
represented. 


The total length of represented time is important within the simulation. 
When the coalition stance was either provide humanitarian aid or secure 
by patrolling, security MOEs were varied at the end of two years 
compared to the end of one year. 


Stances and the interactions of stances between factions have an impact on 
the game’s results. This is more obvious when looking at security, but it is 
true in regards to consent as well. 


In this scenario consent for the Iraqi government is correlated with the 
consent for the coalition. They are not the same, however. This aspect 
was investigated directly and the game represents this well. 


Consent and security are quantitatively different MOEs. They are not just 
scaled differently; it is possible to increase one and decrease the other. 


Player set ROE and force protection can have tremendous implications in 
the scenario. A risk adverse unit with restrictive ROE will fare better in a 
stability focused scenario, especially in regards to increasing security. 


The game is not overly sensitive to the initial settings parameters (these 
are non-scenario dependent assumptions). However, this includes most 
unit attributes (excluding firepower and protection). These attributes have 
marginal effect when changed 20% and only a small effect when changed 
50%. 


One should be very careful when manipulating the mean ROE value and 
the ROE modifier as they have a tremendous impact on the model. 


An increase in a unit’s firepower will decrease consent. Regardless of the 
argument for or against this assumption, it is imperative that unit attributes 
in the settings support the unit’s actual capabilities on the ground. For 
example, most armored units use HMMWV’s in Iraq. Therefore, 
firepower should be reduced accordingly. 


Consent decreased in the two-year model compared to the one-year model 
when the coalition stance was set to provide humanitarian aid, and 
increased when the coalition stance was set to secure through patrolling. 
This shows that even though production increased, the lack of security 
dominated production in determining consent. 
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The developers’ recommendation of a 30-day time step should be adhered 
to. Although changing the length of the time step had minimal effect on 
security, it did result in different values in the consent MOE. 


Throughout this analysis many of the results proved statistically 
significant; however, the results might not be deemed practically 
significant. A 0.2 change in consent could easily be interpreted as 
irrelevant. From the author’s experience with PSOM, this is primarily due 
to the normalizing of the final MOE values and could therefore be 
adjusted to show greater impact from the players’ decisions. 
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VI. CONCLUSIONS 


A. RESEARCH SUMMARY 


This thesis set out to develop and subsequently implement a methodology to 
quantifiably analyze military models, which claim to meet the social implications of the 
modem battlefield. By using a well-vetted realistic scenario, multiple design of 
experiments, various data models, and cumulative statistical analysis the study focused 
on addressing some fundamental questions about the popular simulation model, the Peace 
Support Operations Model. The final results of these experiments and the corresponding 
analysis will provide multiple agencies within the Department of Defense substantial 
insight into PSOM and a solid foundation for further research of PMESII models. This 


chapter concludes the research and analysis conducted within this study. 
B. RESEARCH QUESTIONS 


In developing the purpose of this study we identified four fundamental concerns 


in the realm of PMESII models, into which PSOM falls: 


e Identify the factors which most dramatically influence PSOM’s output. 

e Attempt to assess the accuracy of the Peace Support Operation Model. 

e Make recommendations toward the potential use of the Peace Support 
Operation Model. 

e Conduct a risk analysis of the Peace Support Operation Model. 


This chapter discusses the results of our analysis regarding each of these initial 
issues. Many additional insights were gained through this process and the more 


significant ones are addressed. 
1. Important Factors 


Over 100 factors covering the space of multiple factions, units, population agents, 
settings, and courses of action were explicitly explored in this study. Of significant 


importance are the factors that the players of the game can control. These factors, as they 
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should, have tremendous implications in the simulation’s outcomes. These parameters 
include the units’ stances, rules of engagement level, and force protection level. 
Furthermore, assumptions about the population and the scenario have tremendous 
implications on the responses. Additional factors, which are analogous to the 


simulation’s setup and execution, were identified. 
a. Player Controlled Factors 


Arguably, the most fundamental decision a player representing a faction 
must make is which stance to take. Therefore, it is critical that this decision have an 
effect on the game. We found that the faction stance selection has significant effects on 
the population’s consent to that faction, the consent of that faction’s allies, and the 
security of the country. In addition, the interaction of stances amongst different factions 
proves to be important as well. This also proves true if a faction is acting on behalf of 
another faction. For example, the coalition stances in this scenario often acted on behalf 


of the Iraqi government, resulting in consent for both factions being correlated. 


In regards to stance selection, the behavior of the security MOE seems 
intuitive. When the Sunni Nationalists were in an aggressive stance, the highest average 
security was obtained when the coalition forces took an aggressive stance. However, if 
the Sunni Nationalists are withdrawn, security was at its highest when the coalition was 
either withdrawn or providing humanitarian aid. Because security is a measure of 
perceived violent death, this seems logical. However, it is surprising to learn that 
regardless of Sunni stance, the security values are higher when the coalition took an 
attacking stance as opposed to a securing stance. Nonetheless, because PSOM is very 
malleable, the stances’ parameters can be changed, and we feel this is a simple fix and 


should be looked into further by the developers or users of PSOM. 


With regard to overall Sunni consent toward the coalition and the Iraqi 
government, the stance of the coalition also proved significant. On average, when the 
coalition provides humanitarian aid the consent MOE is higher. The amount of the 
consent values change is also dependent on the Sunni Nationalists’ stance, but the trend 


stays true. By focusing in on Baghdad we found that if the Sunni Nationalists are 
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aggressively attacking the country’s infrastructure, merely providing humanitarian aid 
will do very little for consent. However, with a withdrawn enemy, the coalition’s aid is 
uninterrupted and consent increases. We also found that consent is at its lowest when the 
coalition takes on an attacking stance versus humanitarian aid, securing, or withdrawn. It 
is here that we see the first complicated dynamic in contemporary war; the way to 
increase consent alone seems intuitive—provide aid. This applies to security as well: to 
increase security, provide security. Yet to increase both security and consent takes 
substantial planning on the coalition’s part. The second dynamic is shown when the 
scenario lasts two years as opposed to one year. We found that the positive benefit of 
providing aid when the Sunni Nationalists are in an aggressive stance drops off 
tremendously when the scenario is extended to a second year. It is in this protracted war 
that higher levels of consent can be earned by the coalition by providing increased 


security through offensive stances. 


The rules of engagement level and force protection level the factions 
choose prove very significant toward security, but have minimal effect on consent. The 
results for security were intuitive; the simple analysis is the more restrictive the 
competing factions’ ROE the higher security. The inverse proves true for force 
protection; the greater the risk the factions are willing to take, the lower the security 
level. We also found significant interactions between ROE and stance. For example, a 


faction using an offensive stance with a loose ROE will result in lower security. 
b. Assumptions about the Population 


PSOM is a population centric model. Therefore, it should come as no 
surprise that the initial assumptions made about the population have a tremendous impact 
on the measures of effectiveness. The initial consent level an ethnic group has toward a 
faction will continue to impact the values for consent. Although this is logical, it is a 
powerful initial assumption. Also, the marginal gains of an ethnic group can strongly 
drive the level of consent and security in the simulation. These values represent the 
importance a population agent places on particular goods to include security. They have 


extreme implications on the description of an ethnic group. If these values are entered at 
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too high a level it is almost impossible to increase consent and security will decrease 
significantly. These values must be considered with the utmost care or both security and 
consent will be adversely affected. What is of interest is that the Sunni marginal gains 
for security have the greatest effect on consent where the Sunni marginal gains for goods 


have the greatest effect on security. 


If the simulation is intended for use in any sort of analysis it would be 
imperative to execute the model under multiple initial conditions (Gilbert, 2005). Unless 
of course, the user can find a subject matter expert, for example, that knows the 


importance of sanitation to a Sunni. 
Cs Model Settings 


The settings file focuses primarily on those parameters that are not 
scenario specific, such as the number of personnel in an infantry company or modifiers 
which affect the mathematical models within the simulation. For the most part, these 
settings did not prove overly sensitive to change; however, a few key insights were 


gained. 


The Average ROE is a modifier used primarily in the civilian casualty 
determination algorithm. The developers at DSTL state this value is “normally 3” (Jon 
Parkman, 2008). By varying this value we see tremendous changes in both consent and 
security. This parameter is significant enough to completely change the outcomes of the 
game and should not be adjusted unless completely understood by the user. The other 
settings values which were found significant are a few basic modifiers, primarily the 
population’s memory coefficient and the ROE modifier (see Chapter IV). For the most 


part their manipulation is of minimal effect or easily understood. 


However, there seems to be a lack of effect pertaining to the changing of 
unit capabilities. We found that changing a unit’s (company level) attributes by 20% has 
minimal effect on both security and consent. We then changed these values by over 50% 
from the initial values and saw that large changes in attributes can affect the model. 


These attributes include unit firepower, sensors, ability to change populace attitude, and 
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unit size. The ability to show that these factors matter should assure the user that unit 
capability is important; however, the lack of sensitivity says move cautiously if 


considering the use of PSOM as a force analysis tool. 


The exception to this is the unit’s firepower setting and protection level. 
The protection level acts in an understandable way by having positive changes on 
security and consent; however, firepower has the opposite effect on security and consent. 
When using the preset units in PSOM, it is important to remember that a unit with 
tremendous firepower, such as an armored battalion, might use different equipment based 


on the combat situation. 
d. Time Step and Time of Simulation 


There are two ways to look at time step and both were analyzed in this 
thesis. First, it is clear that the length of the time step is significant. As would be 
expected, the results of twelve 7-day time steps provided significantly different results 
than twelve 30-day time steps. However, we found that fifty-two 7-day time steps 
provided substantially different results than twelve 30-day time steps when looking at the 
consent metric. Therefore, it is best to stay with the developer recommended 30-day time 


step. 


Intuitively, the results of twelve months of game play differ from those of 
twenty-four months of game play with regard to both security and consent. Using the 
standard Iraq scenario, which portrays an aggressive Sunni Nationalist Faction, we see 
that on average security increased from 12 to 24 months. However, when the coalition 
chooses a stance of humanitarian aid, there is a decrease in consent from 12 to 24 months. 
Consent increases on average between 12 and 24 months provided the coalition chooses a 
securing stance against the active insurgency. This implies that PSOM is much more 
than a simple model portraying linear relationships amongst user inputs. The 
implications of non-linear (the curve looks quadratic) changes in consent over time is 
most likely the result of an initial consent for a coalition that provides life improving 


services upfront; however, as the promise of a better country is not met due to a lack of 
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security the consent decreases. Where-as a coalition that initially provides marginal 
goods and services, but provides and then increases security can gradually continue to 


gain consent from the population. 
2. Accuracy of the Peace Support Operation Model 


Because social modeling is far more complex than physics-based modeling there 
is no binary answer as to the accuracy of the model’s outputs. However, through the 
comparison of our quantitative results with doctrine, personal experience, and the help of 
subject matter experts, subjective analysis can be obtained. First and foremost, the model 
uses a multitude of player and subject matter expert inputs to create a simulated 
environment which describes irregular warfare. In the process of conducting this 
research we have explored a vast amount of the PSOM parameter space and found many 
doctrinally analogous results. It is assuring that the model is more than an empty black 
box that provides output regardless of input. Also of note, the model is very flexible. 
There are modifiers that can influence the importance of the majority of the algorithms 
within the simulation. This allows the user to define the space in which the wargame is 


played and therefore influence its accuracy. 


Looking at the consent MOE we have learned that it is very dependent on the 
initial assumptions about the population. We have also learned that it is very difficult to 
increase this factor. Arguably this is logical, as it is very difficult to change the initial 
perceptions of a population. However, what the game does show, which proves a level of 
accuracy, is that a faction cannot just “build” consent. Using the base case Iraq scenario 
with an active insurgency, we manipulated the coalition combat units to all be able to 
provide humanitarian aid in a manner equivalent to humanitarian aid organizations. Over 
a two-year simulation with all coalition units in Sunni populated areas (37 battalions) 
focused on providing an exaggerated amount of aid, there is a clear increase in 
production. However, even under this design, consent either stayed at the initial 
conditions or eventually decreased. However, with the same units now providing 
security, and only non-maneuver units providing aid, we see some excursions where 


consent increased and a slightly higher value in consent than the aforementioned 
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scenario. These results are encouraging in that you cannot just buy a population’s 
consent. This falls into line with the importance of Unity of Effort in Counterinsurgency 
Operations as discussed in FM 3-24 (U.S. Army, 2006). 
Military efforts are necessary and important to counterinsurgency (COIN) 
efforts, but they are only effective when integrated into a comprehensive 
strategy employing all instruments of national power. A successful COIN 


operation meets the contested population’s needs to the extent needed to 
win popular support while protecting the population from the insurgents. 


This goes back to the complexity of the definition of consent. However, along the 
same lines a large part of consent is production and an important part of production is 
determined by the ability to hire human capital. The ability to hire is determined on price 
for labor and consent toward the hiring faction. This price for labor is set, and no faction 
can change this price. This places a great restriction on wealthy factions, such as the 
coalition in Iraq, who can raise the price paid for labor to ensure human capital is 


obtained where needed. 


Another capability PSOM provides is the ability to share the credit for production 
of goods or the provision of security. Throughout this research the coalition shared its 
production credit with the Iraqi government. This did in fact influence the population’s 
consent toward the government which is a key factor in the development of legitimacy 


toward the host nation (U.S. Army, 2006). 


In looking at security we found some effects that should be explored further. 
Security is greatly influenced by the unit’s rules of engagement and risk tolerance, which 
makes sense. However, the stances, although statistically significant, are arguably not 
significant enough. Once again, looking at two years of combat in the base case Iraq 
scenario where there is an active insurgency, we found there to be only a slight difference 
between all coalition maneuver battalions providing aid and all coalition maneuver 
battalions securing through patrols. The mean difference between the two is 0.1, and 
when looking at the extremes in the final DOE where the coalition did its best for security 
while patrolling and the coalition did its worst for security while providing aid, the 
difference is only 0.5. Since security falls within a range of 0 to 10, a difference of 0.5, 


while “statistically significant,” may not appear exceptionally significant for the purpose 
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of analysis. The abilities of stances to affect MOEs can be adjusted within the settings; 
however, this is not a task to be taken on by an individual who does not completely 


understand the model’s algorithms. 


Finally, as discussed earlier, there are questionable implications about the 
absolute changes in unit characteristics. Initially this seems illogical; if you make every 
company in Iraq 20% better you would expect different results in security and consent. 
This can be argued with the recent surge of forces in Iraq. However, the recent success in 
Iraq is much more than a result of increased manpower and capability; it is the result of 
the well-planned use of these increased attributes. It is beyond the scope of this study 
and our current abilities to develop multiple courses of action to test this capability within 


the model. This should be studied prior to using PSOM as a tool for capability analysis. 
3. Potential Uses for the Peace Support Operations Model 


PSOM is a campaign level wargame and therein lies its greatest potential. 
Combat is stochastic in nature (Lucas, 2000), and the addition of a complex populace, 
IGOs, and political players only add to the uncertainty of war. The limited stochastic 
nature of PSOM places a tremendous limit on using a batch mode to analyze scenarios. 
In 2006 the Al-Askari Mosque in Samarra, Iraq was destroyed by Sunni insurgents 
resulting in violence across Iraq. This level of resolution would have to be deliberately 
built into the scenario file in PSOM and would thus be limited to the scenario developer’s 
creativity. However, such realistic actions which can change the state of a campaign can 
be expected to occur with the human-in-the-loop. Without the human players such 
extreme actions would more than likely not happen. As Clausewitz stated: 

They aim at fixed values; but everything in war is uncertain, and 


calculations have to be made with variable quantities 
—Carl Von Clausewitz, On War 


The Peace Support Operations Model provides an opportunity for a large number 
of players to interact and to learn from a simulated population. This population is 
dynamic, consisting of the emersion of many diverse social groups which have the ability 
to change over time. The results of the algorithms, which are constantly being updated 
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and are easily modified, have proven feasible within this study. There is no doubt 


valuable lessons can be learned for use in staff development and training. 


In the same argument, wargaming is of tremendous importance in course of action 
development. If the manpower is available to provide a human-in-the-loop wargame, 
PSOM can provide insight into stability operations where traditional lethality focused 
models cannot. Once a scenario is developed and a plan implemented, it takes minimal 
effort to change the underlying assumptions about the population. For example, a plan 
can be implemented against a multitude of modified populations and results will vary 
accordingly. As long as the decision makers understand that the complexity of social 


modeling ensures no model is a crystal ball, this implementation shows potential. 


As a wargame PSOM provides an environment that brigade and higher staffs can 
easily work within. At the same time PSOM provides a large list of outputs which can 
easily be transformed into strategic level MOEs to parallel current doctrine. These 


attributes strengthen the argument for its use as a wargame. 


The limited sensitivity we found in the model toward unit attributes causes some 
initial concern about the model’s use in force development. Based on our results alone 
the model does not appear sensitive enough to attribute changes. However, as stated 
earlier, these forces were not used in any intelligent manner, so the results are 
questionable. Further analysis on this aspect of the model is recommended prior to its 


use in force capability analysis. 


Also, PSOM proves sensitive to time length and therefore time step. Although it 
is a Peace Support simulation, its use for short-term operations such as disaster relief or 
hasty interventions could be limited. If the needed response resolution from an operation 
is to be measured in days or weeks, PSOM could provide skewed results. The same 
assumption can be made for tactical level operations. The creators of PSOM recommend 
it for task force level use and higher. The simulation’s unit interaction algorithms are 
conducted at a company level resolution. Therefore, further analysis is needed to 
determine if PSOM could be an effective tool for company and below level training and 


course of action development. 
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4. Risk Analysis 


As with all tools in the military, PSOM has inherent risk if misused. PSOM will 
not provide the right answer. It will only provide insights as to what happens in its 
simulated space. And, because it is minimally stochastic, it does not provide a wide 
range of possible outcomes. This is why the human-in-the-loop aspect is essential. Also, 
there are a tremendous number of parameters which can affect the game. This aspect 
makes PSOM very flexible; however, if these parameters are not placed by an expert the 
resulting MOEs are questionable at best. It is absolutely crucial that a scenario builder 
not only know how to change the parameters, but understand the algorithms he or she is 
changing. Currently there are draft manuals for PSOM available which need to be 
completed prior to its mass use (or the scenario builder needs to have the developers on 


speed dial). 


Finally, prior to the use of PSOM as a learning tool or for course of action 
development, the scenario and its settings should be data farmed using a similar, but less 
extensive, methodology to that discussed in this thesis. This will verify the scenario and 
help prevent questionable outcomes from the realm of possibilities creative players bring 
to the wargame. As a participant in wargaming, both as a planner and as a student, 
nothing will shut down the opinion of a military officer faster than infeasible results. The 
data farming process will act as a large-scale rehearsal to ensure the model and its 
scenario are acceptable prior to gathering the expensive amount of manpower required 


for such a wargame. 
Ds Methodology 


Many of the readers of this document are curious about the potential of PSOM. 
However, the analysis of PSOM was partially used to verify the data farming 
methodology as a suitable technique to quantifiably assess military simulation models 
which account for societal phenomenon. Since data farming’s introduction during 
Project Albert in 1999 it has been used repeatedly on a host of agent-based simulation 
packages to gain insight into both the application and the outcomes. Dr. Horne suggests 


its implementation in the verification and validation process (Horne, 2004) and we have 
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implemented it in this thesis with success. By indentifying key factors, developing 
efficient experimental designs, using high performance computing power to conduct the 
experiments, and analyzing simulation output with data mining techniques a wealth of 
information can be gained from a simulation that was designed to be played one step at a 
time over the course of days. To explore even a fraction of the simulated space covered 
over the past six months would be inconceivable if it were to be done via a human-in-the- 
loop game play. Additionally, this was all accomplished with minimal manpower. This 
methodology is scientific in its background and can provide substantial insights into the 


rapidly growing field of PMESII models. 
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APPENDIX A. SCENARIO EXPERIMENTAL DESIGN 


The scenario design of experiment consists of twenty-one factors. The following 
table shows the first five design points for the DOE and all dependent settings used. For 
example, although stance is an experimental factor, the substances must be changed to 


align with the actual stances within PSOM. Factors changed throughout the experiment 








































































































are highlighted. 
DP 1 DP 2 DP 3 DP 4 DP 5 

INGO Stance 8 8 8 8 8 
INGO SubStance 4 4 4 4 4 
Sunni AIF Stance 
Sunni AIF Sub Stance 1 1 1 1 1 
Sunni AIF ATK US 0 0 0 0 0 
Sunni AIF ATK Iraqi Gov't 
(Coalition Stance 8 8 8 8 8 
Coalition Sub Stance 4 4 4 4 4 
(Coalition intel RelationShip with Sunni AIF 0 0 0 0 0 
(Coalition ROE 4 5 5 4 5 
(Coalition Force Protection | 4 2 5 3 
Sunni ROE 2 1 5 4 2 
[Sunni Force Protection 2 3 2 3 1 
[Sunni Political Ideology 13 34 30 6 9 
[Sunni MG Security 0.53 0.375 0.558 0.427 0.347 
Initial Threat 1.2 0.2 -1.4 —0.9 =] 
Initial Consent ~0.1 1 0.2 ~0.9 0.4 
[Normalized Initial Consent 4.750208] 7.310586] _4.50166| _2.890505| 5.986877 
[Sunni Marginal Gains 0.591 0.516 0.422 0.338 0.483 
Coalition Leadership 72 92 58 98 48 
Coalition Experience 55 77 80 91 6 
Coalition Reputation 94 48 88 63 84 
(Coalition Cas Tolerance 19 23 47 8 64 
(Coalition Turns at location 8 4 6 2 7 
HumanAidFactionTargetsandGoodDelivered 0 0 0 0 0 
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APPENDIX B. 


The settings design of experiment consists of sixty-six factors. The DOE shown 


below is the first five design points with the inclusive factor values. 


SETTINGS EXPERIMENTAL DESIGN 


























































































































DP 1 DP 2 DP 3 DP 4 DP5 
UnitFirePower 93.504} 72.352} 68.864 77.6| 86.304 
UnitProtection 99.92} 107.56 94.52 96.12 90.48 
UnitSensor 59.76 58.98 40.4 53.38 55.9 
UNitIntel 22.328) 20.272 22.68} 17.448} 19.632 
UnitPhysicalCamo 1.1608) 1.1068) 0.9076 0.948 1.186 
UnitSocialCamo 0.8228 0.928} 1.0468) 0.8708) 0.9668 
UnitMobilityPerMan 35.352} 28.236} 29.868} 25.788 26.88 
UnitChangeAttitudePerMan 3.2628} 3.2544) 2.7096} 3.4896} 3.3936 
UnitCrime 1.7784, 2.3424) 1.6608) 2.3856) 2.1088 
UnitPolicing 2.9292) 3.0072) 2.9364) 2.7132) 2.6496 
UNitCollateralDamage 7.7588 7.63 6.916} 7.6272) 7.1932 
UNitSize 81 102 98 84 77 
UnitPalitive 0.2729} 0.3341) 0.3144) 0.2506} 0.3366 
StanceAttacksUnit 0.2963} 0.3251) 0.2668} 0.3011) 0.3182 
StanceProtectPopulation 0.4459 0.3736 0.4258 0.3982 0.3394 
StanceProvideAid 0.1688} 0.1663 0.173 0.191 0.189 
StanceModifyPerception 0.4624 0.3634 0.4694 0.468 0.4302 
StanceExtort 0.2715} 0.0747; 0.2289} 0.1206} 0.1293 
StanceCounterCrime 0.2555} 0.2863} 0.3086} 0.3084} 0.2899 
StancelNtelGather 0.5796 0.446, 0.5658) 0.5134) 0.4982 
StanceQRF 0.3246} 0.3494) 0.3755} 0.3341) 0.4696 
StanceAVGSize 30.828} 32.232) 33.204 26.04} 27.084 
StanceProtectionModifier 0.9491) 0.9264) 0.7558) 0.8832) 0.7763 
StanceMobilityModifier 0.9764) 1.0988} 0.8124) 1.1416} 0.9984 
StanceDetectabilityModifier 1.415) 1.4141 1.128) 1.1208) 0.9826 
StanceDetectionModifier 1.4952 1.4526 1.6902 1.4214 1.7004 
StanceRecognitionModifier 1.1948} 0.9812) 0.8716, 0.8448 1.09 
PopulationDecisionRadius 56.28 48.18 48.52 54.84 49.92 
PopulationMemoryCoef 3.546 2.556] 2.9712) 3.4176 3.504 
PopulationConsentPoliticalMScaler | -—0.6954) -—0.9434) -0.9565) -0.865) -—0.6634 
PopulationConsentPoliticalCScaler 0.0855) 0.0807) 0.0933) 0.0979} 0.1185 
PopulationAverageTermInPrison 2.802 2.538 2.76 2.8464 2.5884 
PopulationSelfPresenters 0.5826; 0.5448 0.4596; 0.5152) 0.4714 
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PopulationPoliceClearRate 0.2518} 0.2385} 0.2919} 0.2268} 0.2923 




















































































































PopulationInfectionMargGains 0.2295} 0.2017) 0.2108) 0.1946} 0.2072 
CombatModQueDecayRate 0.0143} 0.0136; 0.0147} 0.0139} 0.0156 
PercentForceonDuty 0.2712} 0.3176} 0.2711) 0.3196} 0.3354 
PlanningDelay 6.5352} 5.9184 6.132} 4.9296) 5.8296 
OperationTime 4.6368} 4.1648) 4.3952) 3.8208) 4.3856 
RecupTlme 10.716 8.98 11 8.62} 11.052 
AvgDistanceTraveled 7.3024} 8.3008} 8.4256} 7.1136 6.432 
MaxFatigue 0.7462} 0.6826} 0.8029} 0.7197} 0.9357 
FatigueDropOff Factor 3.6544) 4.3552 3.712 4.512} 3.4192 
ForceProtectionMean 2.6016 2.6292 3.0324 3.1896 2.4348 
ForceProtectionKValue 0.3547 0.2478 0.3568 0.3288 0.2521 
ForceProtectionMod 0.33 0.2456 0.3475 0.3516 0.3449 
MeanROE 2.8176) 2.9484) 3.5784) 2.5032) 3.2736 
ROEKValue 0.2874} 0.3589 0.261) 0.2636) 0.2696 
ROEMod 0.2778} 0.2612) 0.2896} 0.3532} 0.3272 
MaxLeadershipMod 2.1992 1.972 1.612 1.632) 1.6984 
MinLeadershipMod 0.1615} 0.1602} 0.2021) 0.1613) 0.2214 
LeadershipDropOffFactor 0.0186; 0.0175) 0.0192} 0.0207 0.018 
FamiliarizationStranger 0.5306 0.56, 0.5136) 0.4022) 0.5668 
FamiliarizationNative 1.8528) 2.1352) 2.1632) 2.1408) 1.9304 
FamiliarizationLearniing 0.0982} 0.0904) 0.1146 0.105 0.101 
ExperienceConscript 0.2037 0.2291 0.1912 0.2024 0.1704 
ExperienceVEt 2.244, 1.8736} 1.9704 1.988) 2.2392 
ExperienceLearningFactor 0.0842 0.0965 0.1047 0.0879 0.0752 
InterUnitBaseCasATT 0.0872} 0.1044) 0.0875} 0.1007} 0.1036 
InterUnitBaseCasualtiesDEF 0.1082} 0.1086} 0.0905} 0.1044) 0.0852 
InterUNitBaseContactSlze 29.292) 26.076 26.52} 29.844) 24.348 
GoodExpected 0.4174} 0.4004 0.593} 0.5432} 0.5496 
GoodProtectionValuePower 10880 8636 10856 11092 11624 
GoodProtectionValueWater 64.344 77.812 56.364 83.58 74.732 
GoodProtectionValueEducation 63.364 76.02 71.54, 65.184) 83.664 
GoodProtectionValueHealthCare 132.84, 144.54) 166.08) 124.56) 172.62 
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APPENDIX C. CUMULATIVE EXPERIMENTAL DESIGN 


The first five design points of the cumulative experimental design are shown 


















































below. 
DP 1 IDP 2 DP 3 IDP 4 IDP 5 

Coalition Stance 0 0 0 0 0 
Simulation Time 60 60 60 60 60 
Unit Fire Power 81 56 63 69 125 
Unit Protection 150 75 94 113 144 
Unit Sensors 131 138 56 81 94 
Unit Change Attitude 4 6 3 10 2 
Unit Manpower 75 50 113 106 81 
Unit Intelligence 48 29 44 24 20 
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APPENDIX D. CORRELATION AND SPACE FILLING QUALITY 


OF FACTORS 


This appendix shows the pair-wise correlations and space filling qualities for the 


quantitative variables for each DOE. The use of NOLHs results in low pair-wise 


correlations while maintaining good space filling properties. 


A. SCENARIO EXPERIMENTAL DESIGN 








‘Correlations 

Coaltion ROE Coaltion Force Protection Sunni ROE Sunni Force Protection Sunni Poltical ldeology Sunni MG Securty Initial Threat itil Consent Sunni Marginal Gains Coaltion Leadership Coaltion Experience Coaltion Reputation Coaltion Cas Tolerance Coalition Turns at location 
Coaltion ROE 4.0000 0.0000 0.0000 0001 0.0081 0002 = 00008 = 0.0074 0.0054 0.0056 (1.0085 0.0076 0,008 0.005 
Coaltion Force Protection 0.0000 1.0000 -Q,0000 010000 010082 0.0088 «0.0047 0.0088 0148 00123 (10064 0.0057 0.0038 00173 
Sunni ROE 0.0000 0.0000 1.0000 00000 00028 O93 0005 = 0.0025 0.0024 00037 0.0012 008 0.0032 0.0 
Sunni Force Protection 0.0001 0.0000 0.0000 ,0000 0.0019 0.0033 0,008 0.0088 01,0068 0.0064 0.0014 0.0082 0.0042 0.0008 
Sunni Poltical ideology 0.008 0.0032 0.0028 00019 1.0000 00017 00045-00087 0,008 0.0042 0.0044 0.0056 0,000 ooo? 
Sunni MG Securty 00012 0.0088 00183 0.0038 00017 {ooo 0.00% == 0.0017 00010 00019 0.0081 0012 0.0070 Qi 
Inia! Threat 0.0006 0.007 0.0051 0.0069 0.0065 00095 1.0000 0.0033 0.0201 0.0064 0.0069 0.0049 0.0089 00048 
Initial Consent 0.0074 00036 00025 0.0038 0.0037 Ooo? = 00083 = 4.0000 0,086 ond? 0.0005 0.0151 (0,083 00026 
Sunni Marginal Gains 0.0054 0.0148 0.0024 0,088 0.0008 0.010 00201-00086 ,0000 00183 0.0037 0.0075 00182 0.0035 
Coaltion Leadership 0.0056 00123 0.0037 0.0084 010042 00019 00064 O14? Ones 1.0000 0.0067 0.0098 0.0039 00244 
Coaition Experience 0.0085 0.0064 0.0012 00014 0.0044 Ooo = 00088 -0.0005 0.0087 0.0067 {0000 0.0090 00020 00008 
Coaltion Reputation 0.0076 Os? 00118 0.0082 0.0056 0.0122 0004900151 0.0075 0,098 0.0080 1.0000 0.0014 0.0211 
Coaltion Cas Tolerance 0.0008 0.0038 0.0032 0.0042 0.0000 007000089 0.0083 0.0182 00039 0.0020 00014 4.0000 00099 
Coaltion Turns et location 00015 A073 0.0190 0.0008 On07 0077 00048 = 0.0026 00035 010244 0.0009 0021 0.0039 4.0000 
HumanAidFactionTaryetsandGoodDelivered 0.0000 0.0000 -0.0000 0.0000 0000 00000 = 00000» 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0000 


Pair wise correlations for the quantitative factors in the scenario DOE. The largest 


absolute value is .02. 


‘Scamerplot Mawix 
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Scatter plot with factor names on the diagonal of the Scenario DOE. This scatter 


plot displays the space filling quality of the NOLH for this experiment. 


SETTINGS DOE 


B. 


Because of the number of factors and the number of design points, the correlation 


matrix for the setting DOE cannot be displayed in this appendix. However the greatest 


correlation was .0025, so clearly our results qualify as “nearly orthogonal”. Below is the 


Its purpose is to display the space 


filling power of COL Hernandez’s (2008) expansion of the NOLH. 


scatterplot matrix. This figure is clearly illegible. 
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C. CUMULATIVE DOE 








Correlations 
Sunni MG Security Sunni Marginal Gains UnitFirePow er UnitProtection UnitSensors — UnitSize UnitChangeAttitudePerMan —_Unitintel 
Sunni MG Security 1.0000 -0.0001 0.0095 -0.0024 0.0085 0.0177 -0.0172  -0.0006 
Sunni Marginal Gains -0.0001 1.0000 -0.0060 -0.0065 0.0002 0.0037 0.0532 0.0017 
UnitFirePow er 0.0095 -0.0060 1.0000 -0.0000 -0.0000 0.0061 -0.0331 0.0048 
UnitProtection -0.0024 -0.0065 -0.0000 1.0000 0.0000 = -0.0112 -0.0167 0.0072 
UnitSensors 0.0085 0.0002 -0.0000 0.0000 1.0000 -0.0010 -0.0029 -0.0106 
UnitSize 0.0177 0.0037 0.0061 -0.0112 -0.0010 1.0000 -0.0127 = -0.0073 
UnitChangeA ttitudePerMan -0.0172 0.0532 -0.0331 -0.0167 -0.0029 -0.0127 1.0000 0.0102 
Unitintel -0.0006 0.0017 0.0048 0.0072 -0.0106 -0.0073 0.0102 1.0000 

Above are the pair wise correlations for the quantitative factors in the scenario 


DOE. The largest absolute value is .05. 





{ Scatterplot Matrix 
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This scatter plot with factor names on the diagonal of the Cumulative DOE 


shows the space filling quality of this design of experiment. 
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APPENDIX E. HUMANITARIAN AID SETTINGS 


In order to explore the consent metric, we gave all maneuver battalions 
humanitarian aid providing capabilities. The below screen shot shows the attributes we 


gave these units. 


Data and Settings Editor 


Data and Settings 


: — a / 
| Save/Load Settings | Unit Abilities stances | Goods Details | Descriptors | Population Agents | Pre-Set Unit Types | Terrain | Display Colours | Combat Modifiers | Outputs | Weather | 


i . Oe Add Ur Remove 
sca | = = vj eas 


Military Yalues Reconstruction Values 


Name: larmd Sqn (cR2) 














Palliative 








Domain: Land ] Mobility: |30 


Firepower; |80 Change Attitude: (3 





Protection; 100 Crime: 


Sensors: [50 c Policing: \3 


iF 
1 
1 
|1 
1 
1 
\1 
iq 
l1 
1 
1 
1 
1 
i 
Fr 





Intelligence: f Collateral [5 
. Damage: | 


Social = S, 
Camoflague; Domain Modifiers 


Physical 
Camoflague: 


Land: | 1 


Manpower: 


air: |0.2 
Max Footprint. =} cE - + — 
Size: 








Maritime: (0 


Logistics Cost: | 





Logistics Provision: { 


Save Changes Cancel 
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